First of all: Thank you for the MM debug version and the inside informations.
The latest rescan for all my songs took 44 minutes.
(from start scanning until the point where MM seems to be frozen with the last "added" song shown in the taskbar,
that's the point, where it takes hours until MM returns to normal working state***)
The following should be noted:
Every step you do works fine. At least for a few hundred or a thousand tracks.
But please think bigger. What about hundreds of thousands or even millions of tracks?
Thanks to the DebugView log I would like to suggest some improvements:
Example:
For every single track you query the MM.DB with
SELECT Songs.* FROM Songs WHERE Songs.IDMedia=? and Songs.SongsPath=?
Even on a very fast system this takes - according to the DebugView-Output -
about 450-600microseconds, or 0.5 miliseconds or 0.0005 seconds.
Again: This is not a problem for a few hundred songs. But for millions?
If you calculate 1 Million tracks you need 1'000'000 x 0.0005 seconds = 500 Seconds = 8,3 Minutes only for this query
And this query is only one of many processed for every track.
I suggest to rethink this process.
Step #1:
Assuming that most of us has some structure for the songfiles,
we could think about something like a folder or an album.
Why not read a whole folder from disk with something like "Directory.EnumerateFiles"?
You make only one call per folder and get all the full names (including paths) for the files in the specified directory.
Then you do the same with only one query with the specified foldername in the MM.DB with
SELECT Songs.* FROM Songs WHERE Songs.IDMedia=? and Songs.SongsPath=FolderName
After matching the songs in memory you have two kind of entries
a) the ones found in DB = all this songs already exists in DB
b) the ones not found in DB = new or changed ones
For b) you have to scan every new file to get the additional tag information. The way it works today.
For a) you only have to scan the files with changed "last modified" timestamp (according to the user settings).
BTW: I don't know which programming language do you use. But with the help of something like this
https://sourceforge.net/projects/fastfileinfo/
original source was:
https://www.codeproject.com/Articles/38 ... Enumerator
you can speed up things dramatically.
Step #2:
CREATE TEMP TABLE ScannedSongs_2 ( IDSong INTEGER )
DB exec SQL: INSERT INTO ScannedSongs_2 (IDSong) VALUES (?)
I'm not sure if this table is created in the MM.DB or just as memory table.
Anyway: I would suggest to use transactions for all the INSERT INTO's and not execute it for every single track.
(The same way you do it already to UPDATE Songs SET SongPath=?,Artist=?....)
Step #3:
AddToDatabaseRec: Number of cover candidates: 0
AddToDatabaseRec: Going to save covers
I'm not sure what exactly you are doing to decide to get 0 cover candidates.
In my case I have deactivated the setting "search for covers in the same directory".
So it should not search at all for covers especially not with an additional "file search"- or "cover save"-call.
Step #4:
Prgrs: Updating previous level from: 0 by 10
Prgrs: Updating previous level from: 0.3 by 0.7
Prgrs: Updating previous level from: 510 by 10
Prgrs: Creating a new level.
Prgrs: Deleting a level.
I'm not sure what exactly you are doing while "Updating previous level"
If it is about the volume level per Track or Album:
In my case I have deactivated all the volume leveling settings.
This is confirmed by several "TDevices.disableDiscVolume" log entries.
***
Final words to the endless last step (after rescanning the songs, until the GUI responds again):
MM is actually writing endless lines of
Thread '_ManageTransactions' has started/has finished with 4 or 5 lines of
DB open SQL: SELECT * FORM Songs WHERE Songs.ID=?
5 lines needs about a second.
I assume, you will do that for all tracks.
1'000'000 : 5 = 200'000 seconds = 3'333 minutes = 55,5 hours...
Why just using 2 Threads from all 32 available?
Do you really write only 4 or 5 songs with a single transaction?
First of all: Thank you for the MM debug version and the inside informations.
The latest rescan for all my songs took 44 minutes.
(from start scanning until the point where MM seems to be frozen with the last "added" song shown in the taskbar,
that's the point, where it takes hours until MM returns to normal working state***)
The following should be noted:
Every step you do works fine. At least for a few hundred or a thousand tracks.
But please think bigger. What about hundreds of thousands or even millions of tracks?
Thanks to the DebugView log I would like to suggest some improvements:
Example:
For every single track you query the MM.DB with
SELECT Songs.* FROM Songs WHERE Songs.IDMedia=? and Songs.SongsPath=?
Even on a very fast system this takes - according to the DebugView-Output -
about 450-600microseconds, or 0.5 miliseconds or 0.0005 seconds.
Again: This is not a problem for a few hundred songs. But for millions?
If you calculate 1 Million tracks you need 1'000'000 x 0.0005 seconds = 500 Seconds = 8,3 Minutes only for this query
And this query is only one of many processed for every track.
I suggest to rethink this process.
Step #1:
Assuming that most of us has some structure for the songfiles,
we could think about something like a folder or an album.
Why not read a whole folder from disk with something like "Directory.EnumerateFiles"?
You make only one call per folder and get all the full names (including paths) for the files in the specified directory.
Then you do the same with only one query with the specified foldername in the MM.DB with
SELECT Songs.* FROM Songs WHERE Songs.IDMedia=? and Songs.SongsPath=FolderName
After matching the songs in memory you have two kind of entries
a) the ones found in DB = all this songs already exists in DB
b) the ones not found in DB = new or changed ones
For b) you have to scan every new file to get the additional tag information. The way it works today.
For a) you only have to scan the files with changed "last modified" timestamp (according to the user settings).
BTW: I don't know which programming language do you use. But with the help of something like this
https://sourceforge.net/projects/fastfileinfo/
original source was: https://www.codeproject.com/Articles/38959/A-Faster-Directory-Enumerator
you can speed up things dramatically.
Step #2:
CREATE TEMP TABLE ScannedSongs_2 ( IDSong INTEGER )
DB exec SQL: INSERT INTO ScannedSongs_2 (IDSong) VALUES (?)
I'm not sure if this table is created in the MM.DB or just as memory table.
Anyway: I would suggest to use transactions for all the INSERT INTO's and not execute it for every single track.
(The same way you do it already to UPDATE Songs SET SongPath=?,Artist=?....)
Step #3:
AddToDatabaseRec: Number of cover candidates: 0
AddToDatabaseRec: Going to save covers
I'm not sure what exactly you are doing to decide to get 0 cover candidates.
In my case I have deactivated the setting "search for covers in the same directory".
So it should not search at all for covers especially not with an additional "file search"- or "cover save"-call.
Step #4:
Prgrs: Updating previous level from: 0 by 10
Prgrs: Updating previous level from: 0.3 by 0.7
Prgrs: Updating previous level from: 510 by 10
Prgrs: Creating a new level.
Prgrs: Deleting a level.
I'm not sure what exactly you are doing while "Updating previous level"
If it is about the volume level per Track or Album:
In my case I have deactivated all the volume leveling settings.
This is confirmed by several "TDevices.disableDiscVolume" log entries.
***
Final words to the endless last step (after rescanning the songs, until the GUI responds again):
MM is actually writing endless lines of
Thread '_ManageTransactions' has started/has finished with 4 or 5 lines of
DB open SQL: SELECT * FORM Songs WHERE Songs.ID=?
5 lines needs about a second.
I assume, you will do that for all tracks.
1'000'000 : 5 = 200'000 seconds = 3'333 minutes = 55,5 hours...
Why just using 2 Threads from all 32 available?
Do you really write only 4 or 5 songs with a single transaction?