Google sync - how do you fix bad uploads? [#15849]

Report bugs & feature requests for MediaMonkey 5 and learn about the newest builds.

Moderator: Gurus

Linwood
Posts: 48
Joined: Sat Jul 13, 2019 6:49 pm

Re: Google sync - how do you fix bad uploads?

Post by Linwood » Tue Jul 30, 2019 6:58 pm

To the original subject matter of fixing a bad upload -- with the options set as described above, I deleted an uploaded track on GPM, then did a permanent delete on GPM, and did a sync. It correctly and quickly found that and uploaded.

So my original concern seems to have been a bad setting of options on my part, it does allow you to delete and re-upload to GPM a track that may be bad or otherwise a mistake.

However, I then did tagging on an album with XX tracks, adding a cover art and various changes to titles, artists, etc. I did a sync, and it said

Updated 1 files, did not import 4 files.

The one file is the one I did an upload of a bit ago (correct), the four are the same four it keeps skipping. It did not show changes due to the tagging.

I think that may be wrong also -- should not changes in the track from tagging result in being sync'd back to Google? Note that under "remote content" I do not have "Sync GPM metadata with the library" because I do not quite understand it yet, and think it may bring GPM back to MM5 -- it is possible though that kept this album from being updated on GPM. Did it?

Barry4679
Posts: 763
Joined: Fri Sep 11, 2009 8:07 am
Location: Australia
Contact:

Re: Google sync - how do you fix bad uploads?

Post by Barry4679 » Tue Jul 30, 2019 8:53 pm

Linwood wrote:
Tue Jul 30, 2019 6:58 pm
I think that may be wrong also -- should not changes in the track from tagging result in being sync'd back to Google? Note that under "remote content" I do not have "Sync GPM metadata with the library" because I do not quite understand it yet, and think it may bring GPM back to MM5 -- it is possible though that kept this album from being updated on GPM. Did it?
I guess that I shouldn't answer this, because I am only a few steps ahead of you ... but Ludek is in deep night ATM, so ...

You have disabled tag sync, in either direction, by turning that option off.

To ensure that your local tags do not get polluted, you should leave it turned off. Ludek said that, in a coming version, he will default the control to "off", but make it control tag sync in a GPM->Local direction only ... from this point you should see that your local tags are sync'd to GPM.

[UPDATE] The current situation is (I think), that a sync with the option off, means that a sync handles track existence only ... ie. it makes sure that both location have the same tracks ... and where it manages to match previously unmatched tracks, it will store the GPM track location into MM

I am not sure yet which tags are synchronised.

And I am not dead-set sure yet what triggers the sync ... Ludek seems to say that, in a two direction sync situation, that a sync will be triggered by a change in either location ... which sounds clever ... I don't know how he handles collisions

Ludek
Posts: 2999
Joined: Fri Mar 09, 2007 9:00 am

Re: Google sync - how do you fix bad uploads?

Post by Ludek » Wed Jul 31, 2019 6:10 am

As for the number of files in GPM: Previously it could show less because of the errors you got during uploading? i.e. you had a transient errors based on the logs so some tracks were uploaded on the next sync?

As for the number of library tracks: 3322 files is probably the number of all songs in your entire library, i.e. not only the Music collection that has 3292 files?
Check the others collection or enable the 'Entire Library' node to view all the tracks.

Re skipped files: They are mostly skipped based on your import criteria.
e.g. Options > Library > [x] Ignore files shorter than 15 seconds
Or the checkboxes on the GPM profile config like '[x] Only include convent that matches files already in the database' ?

Neverhteless you can click the [View log] button to see which files were skipped and then use the [Save..] button to save the log for post analyzis.

Linwood
Posts: 48
Joined: Sat Jul 13, 2019 6:49 pm

Re: Google sync - how do you fix bad uploads?

Post by Linwood » Wed Jul 31, 2019 8:59 pm

Ludek wrote:
Wed Jul 31, 2019 6:10 am
As for the number of library tracks: 3322 files is probably the number of all songs in your entire library, i.e. not only the Music collection that has 3292 files?
Check the others collection or enable the 'Entire Library' node to view all the tracks.
Bingo. I had 10 audiobooks and 20 videos in there which I was not even aware of (came in from an iTunes library).

Though this does beg the question why I saw the number 3322, since during the setup of the items to sync I ONLY selected music.

I'd also like to suggest that EVERY list show record counts. For example, once I found Entire Library (had no idea it was available, I really need to find and read the manual :lol: ), I don't see any row counts on any of the screens, e.g. right now staring at All Tracks, and nothing at the bottom. Change to Music, All Tracks, and count is there.
Ludek wrote:
Wed Jul 31, 2019 6:10 am
As for the number of files in GPM: Previously it could show less because of the errors you got during uploading? i.e. you had a transient errors based on the logs so some tracks were uploaded on the next sync?
Well, yes, but trying to track those from the various numbers presented makes no sense, because the numbers do not seem drawn from a consistent place (e.g. some places it used 3322 above and some 3292 apparently).

Here's an example right now... EVERY file is uploaded to GPM. 3292. It's the right number for music.

When I go into Devices, GPM, Remote, All Tracks, it shows 3292 BUT there are 15 files that have the local storage indicated not the cloud. That would SEEM to be saying there are 15 still waiting to go. When I hit Sync Now, it says zero updated, zero imported, Library has 3322 (ok -- I get where that comes from, but it seems wrong in context) and "Did not import 4 files".

When that dialog is closed nothing else happens -- it does not try to sync the 15 files. At one level that is correct -- they are in fact on GPM so it does not need to. But why does it continue to show them as without a cloud symbol yet know it does not need to sync?

You ask about not meeting the criteria -- what criteria? I am unclear where I look for that with respect to this? I picked the first one. It is Julian Bream, Drei Tentos: No. 1. Tranquilmente ''Du schönes Bächlein', it is a flac type file of length 2:25 under Music. Nothing jumps out to me as to why it would not go.

As to the 4 files that say "skipped", all four (but only 4) of them are on the list of 15 that do not show the cloud symbol, but I see nothing unusual about them. Some are flac, some m4a, all are music all have typical length (only two under a minute, those 44 and 57 seconds).

I realize it may seem wrong to complain that I can't balance these numbers when I know I have all songs loaded -- I get that. But in the sense of providing feedback on what users might encounter, assume for a moment they did NOT all upload, and I was looking at these numbers which would not reconcile.

The data really needs to be clearly defined when you filter in these ways. Maybe there is a document somewhere which defines these displays and I simply do not understand. But to me it is very confusing.

Ludek
Posts: 2999
Joined: Fri Mar 09, 2007 9:00 am

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Ludek » Sun Aug 04, 2019 1:46 pm

When I go into Devices, GPM, Remote, All Tracks, it shows 3292 BUT there are 15 files that have the local storage indicated not the cloud.
Do you mean Devices & Services > GPM > All tracks ?
There are only tracks with cloud symbol only, link+cloud, cloud+hdd.
[cloud+hdd] means that the cloud track is both in GPM + downloaded/paired with a local copy.
[cloud+link] means that the cloud track is included in your local database, i.e. can be streamed from GPM, but has no local copy
[cloud] mean that the track is only in the cloud library (GPM), but not in your local database/library

So if there are 15 files that have the local storage indicated not the cloud then it sounds rather like a bug.
Could you attach screenshot of those 15 files ? And screenshot of the checkboxes on Devices & Services > GPM > Remote content page ?


EDIT: Re the legend (per suggestions in https://www.ventismedia.com/mantis/view.php?id=15023 )
[drive][cloud] - cloud track that has been matched with already existing local track on hdd
[download][cloud] - cloud track that has been downloaded to library (cached)
[link][cloud] - cloud track presented in library (added into library), but not cached
[cloud] - just cloud track (e.g. in Devices & Services > GPM > All tracks) - not in library
[drive] - just local track scanned into library (as in MM4)
[link][youtube] - youtube track in library
[youtube] - just youtube track (not in library)

Linwood
Posts: 48
Joined: Sat Jul 13, 2019 6:49 pm

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Linwood » Mon Aug 05, 2019 11:05 am

Ludek wrote:
Sun Aug 04, 2019 1:46 pm
When I go into Devices, GPM, Remote, All Tracks, it shows 3292 BUT there are 15 files that have the local storage indicated not the cloud.
Do you mean Devices & Services > GPM > All tracks ?
Right now I am in MM5 and here's what I see:

Music, All Tracks, shows 3292 files.
16 files show HDD only
3276 files show HDD + Cloud

Entire Library, All Tracks, shows 3322 files.
46 files show HDD only
3276 files show HDD + CLoud

Devices & Services, GPM, Remote content, All tracks shows 3292 files.
16 show HDD only
3276 show HDD + Cloud

GPM itself shows 3292 files, every single one uploaded by MM5, it was empty before.

So I did this: I deleted one song from GPM itself from the web site, then did sync.

Dialog pops up with scan results:
Library has 3322 files.
Updated 1 file (the one I deleted)
Did not import 4 files

The 1 makes sense, the 4 is also part of the 16 (If I look at the 16 I see them). But I'm not sure what the 16 really mean, it appears there are 16 files MM5 uploaded, but then lost track of?

If it's useful I can wipe the whole thing in GPM and start over?

Ludek
Posts: 2999
Joined: Fri Mar 09, 2007 9:00 am

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Ludek » Tue Aug 06, 2019 4:21 am

OK, I think that I simulated your issue and found a way how to replicate.

I used this steps:
1) unchecked the '[ ] Scan GPM content... ' checkbox
2) selected a file that is not in GPM yet > right-click > Send to > GPM
3) go to D&S > GPM > All tracks
=> the lastly uploaded file appears with the hdd icon only (is missing the cloud icon)
4) Checked the '[x] Scan GPM content... ' checkbox > [Apply] > [Sync now]
=> the file gets the cloud icon as expected

I'll fix this for build 2191 (tracked as https://www.ventismedia.com/mantis/view ... 023#c51355 )

Linwood
Posts: 48
Joined: Sat Jul 13, 2019 6:49 pm

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Linwood » Wed Aug 07, 2019 10:54 am

So let us start with a clean slate and do a full upload. With 2190 released I did the following. Sorry, this is long, but I think following the process may provide clarity if you can make it through all this length...

Deleted device in MM5
Deleted all songs on GPM
Added device back to MM5 and authorized
Added all music (3292) in MM5 in what to sync
Checked boxes Scan GTM to local, Only include that matches
Not checked Download GPM and not check Sync Metadata
Apply, sync
  • Library has 3322 files (There are 30 that are not music)
    Imported 0, updated 0, etc.
    Syncing 3292
Started at 10:08pm
Finished about 10:45am so about 12 hours
GPM itself only shows 3264 songs

When it finished (or more precisely stopped), it had one error on a large popup box:

Problem while copying Michael Beghiat \ Transformation \ 03 Michael Beghiat - Sun & Moon.mp3
Cannot read property ResponseCode of undefined

That is indeed not on GPM, but also 28 songs are not in GPM either whereas the error dialog shows only one. When I dismissed that one, hidden behind it was yet another popup saying "There was an error while connecting" with the same error but that did not show the track. Same error message though, so this seemed redundant.

What I think actually happens is one of those boxes (the first I saw, which was on top) was intended to be a summary on completion, if there were multiple errors. The second I think indicates a premature termination and why. I say premature because 28 songs were not uploaded (from GPM's count, many more from MM5).

I find the boxes confusion -- maybe they need a better title, definitely the premature termination box should come up first, clearly indicate it was a premature termination, and then and only after you dismiss it, provide a the summary box with any son problems?

That presumes it is even possible to have more than one problem without termination.

But it gets more confusing:

Afterwards (before another sync), Music, All Tracks showed only HDD symbol, no cloud symbols at all. I guess those get updated only on a second sync? Or was it because this ended prematurely?

Clicking on GPM device, remote content All Tracks I got 2788 files shown, only HDD. 2788 is quite different from 3264 that GPM has, that seems to be a problem.

Clicking on sync where it scans the remote it says "x of 2788" got to 2787 and stopped (the 2787 vs 2788 is a bit confusing but maybe that is the one it aborted on?).

Popup says: 3322 in library, imported 0, updated 2784, did not import 4 (same 4 I think as I keep having trouble with). I can't reconcile this 2784 with GPM's actual 3264.

As another comment on terminology: the term "import" here is vastly confusing to me since I am not trying to import anything. I also do not know what the implications are of failing to "import" those four, and more relevant if they were skipped due to some criteria -- what criteria? How would I "fix" this, if even it needs to be fixed?

After acknowledging the popup, Sync continued with 20 (+/-, it did not seem to start at 1) of 504.

The 504 it shows is correct in relation to the 2784, but I can't reconcile 2784 with the 3264 actually in GPM (remember it was absolutely empty when I started this -- including trash). It seems to have lost track of a large number of files it previously uploaded, even though at this point it has done a scan of GPM itself.

So I let it finish after dismissing that dialog.

There was no popups at all, presumably everything worked.

If I got to Music, All Tracks, it shows 3292 tracks, of which 517 have just a HDD symbol, and 2775 have a cloud. I can't figure out where either one of those last numbers come from relative to whst it had before.

If I go to Devices, GPM, and click All Tracks, it does a scan quickly and shows exactly the same 517 with HDD, but it shows 2766 for a cloud symbol, down 9 from Music's All Tracks. The 'real' GPM at this point shows all 3292, so all songs are there. Again, not really able to reconcile this even if +/-4 is thrown in.

So I hit Sync Now again.

It does a "scanning" of 3283, then the popup says:
Library has 3322
Updated 504
Did not import 4

It then did not upload any new to GPM, and finished immediately (correct).

Music, All tracks, shows 24 files with just HDD, 3268 with Cloud and HDD (3268 + 24 = 3292 so correct in total but confusing individually)

GPM, All tracks, shows the same.

In summary:

The good news is I think all the songs are there. The other good news is if you just set it to sync automatically and ignore it a few times, all the songs would get there. (Although I wonder -- does that popup dialog prevent further sync activity until dismissed? So you cannot just set it to repeatedly try? I did not attempt this myself).

The bad news is if you are trying to watch what it does and tell how far along it is -- you are going to get very confused.

I think providing all these interim counts is a great thing. Being able to summary with column filters is great also. It gives reassurance that all is well. But the flip side of that is that if you cannot tell what the numbers mean, you start to suspect MM5 is not working properly.

If you believe the numbers are right, they need better labels and some kind of explanation or reconciliation.

If you agree they are not right... well, that's why it is a beta. :)

Ludek
Posts: 2999
Joined: Fri Mar 09, 2007 9:00 am

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Ludek » Thu Aug 08, 2019 12:05 pm

Afterwards (before another sync), Music, All Tracks showed only HDD symbol, no cloud symbols at all. I guess those get updated only on a second sync? Or was it because this ended prematurely?
As I wrote above. This is fixed in build 2191.

As for the "premature termination" -- that's a pity that you haven't caught this in a debug log?

I am going to do more sync testing myself to see whether I can replicate those issues.

Ludek
Posts: 2999
Joined: Fri Mar 09, 2007 9:00 am

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Ludek » Thu Aug 08, 2019 1:22 pm

Did not import 4
I guess that this always reproducible if you press [Sync now] even without deleting anything from GPM?

If yes, could you catch it in debug log (item 4b here: https://www.mediamonkey.com/forum/viewt ... 30&t=86643 ) ?

Linwood
Posts: 48
Joined: Sat Jul 13, 2019 6:49 pm

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Linwood » Thu Aug 08, 2019 3:57 pm

Ludek wrote:
Thu Aug 08, 2019 12:05 pm
Afterwards (before another sync), Music, All Tracks showed only HDD symbol, no cloud symbols at all. I guess those get updated only on a second sync? Or was it because this ended prematurely?
As I wrote above. This is fixed in build 2191.

As for the "premature termination" -- that's a pity that you haven't caught this in a debug log?

I am going to do more sync testing myself to see whether I can replicate those issues.
Re 2190 vs. 2191 (blush) sorry, I did not read carefully enough.

I am afraid to try to leave a debug log running over a 12 hour upload, which seems to be what is required for it to abort. Putting in the forced abort did not yield anything interesting. Though I guess debug view will log to a file, maybe that would be OK, just didn't want everything to crash from running out of memory.

Maybe when 2191 comes out I'll redo the whole test and get a full log from it. Would that be helpful in the termination issue? Then I could send the last few hundred lines or some such?

Ludek
Posts: 2999
Joined: Fri Mar 09, 2007 9:00 am

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Ludek » Fri Aug 09, 2019 7:08 am

Ok, but I guess that also now (when everything is in GPM) and you click [Sync now] then it proceeds immediatelly, but the scan log reports did not import 4 files (did not match import criteria), isn't this the case? If yes, could you attach debug log of that?

Linwood
Posts: 48
Joined: Sat Jul 13, 2019 6:49 pm

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Linwood » Fri Aug 09, 2019 7:18 am

> 4 files (did not match import criteria), isn't this the case? If yes, could you attach debug log of that?

Done. Wasn't sure how to attach here so it's a ticket at #CHY-281-69913.

Also not at all sure why there were 9 songs updated. I am not aware of touching those songs in the interim. But I do not see that as having done any harm either.

Ludek
Posts: 2999
Joined: Fri Mar 09, 2007 9:00 am

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Ludek » Fri Aug 09, 2019 2:13 pm

Thanks, I've got the log and found the reason for these bugs.

I am going to fix this for 2192+

Thanks so far and have a great weekend!

Linwood
Posts: 48
Joined: Sat Jul 13, 2019 6:49 pm

Re: Google sync - how do you fix bad uploads? [#15849]

Post by Linwood » Fri Aug 09, 2019 3:12 pm

I'll watch for when you have a version with everything so far fixed, and then wipe and redo from scratch and see what I get.

And log to a file.

Post Reply