Last.fm Playcount Import (2010-03-28) [MM3/4]

[Psyker7]

This script sync's last.fm playcounts and last played dates into MM.
It will find total playcounts of every track in your last.fm history, check if it is higher than the value in the database (for all tracks of that title/artist - even if multiples) and updates it if so.
It will also update the last played times for all tracks with a newer last played on last.fm than in your MM database.

Get it from: http://psykar.com/scripts
Direct link: http://psykar.com/script-files/LastFmImport-3.3.mmip

It's been extensively tested now, but if you run into issues, sending me the log file would be a great help! %appdata%\MediaMonkey\Scripts\LastFmImport\Log.txt

Notes:

• Does *not* update MM's internal play histories. Feel free to use other scripts to fix this up (unlikely to be added as a feature here) http://www.mediamonkey.com/forum/viewto ... =2&t=31809

Source code is up at https://github.com/Psykar/MM-LastFmImportPython
The tree needs cleaning up but figured I'd shove it up anyway given I'm now distributing an exe so you can look at the source code. It's compiled with pyinstaller.

Version history is here:
https://github.com/Psykar/MM-LastFmImpo ... its/master

It will find total playcounts of every track in your last.fm history, check if it is higher than the value in the database (for all tracks of that title/artist - even if multiples) and updates it if so.
It will also update the last played times for all tracks with a newer last played on last.fm than in your MM database.

Useful if you lose your play counts, or you have a portable player (such as iPhone) which will scrobble tracks to last.fm OK, but won't update playcounts in MM.

A log file is created automatically containing all tracks which were updated in a tab delimited file in your %appdata%/mediamonkey/lastfmimport directory.

A first run over a database can a few hours, but interrupting the script is safe, and running it later will pickup where it left off.
To speed it up somewhat, turn *OFF* autorate accurate, otherwise every track updated will need to be written too as the script processes.


======

My first real scripting attempt for MM, thanks to:
Teknojnky - author of last.fmnode - for code for various functions, hacked a LOT.
Trixmoto - for mmip packaging of normal scripts, had me stumped for a while there

======================================================
Download:
http://psykar.com/scripts/
======================================================

I VERY STRONGLY recommend a database backup before you try this.


ToDo:
- Fuzzy matching

[trixmoto]

Sounds like this will be a useful script for many people. I'm glad my MMIPs helped you with yours. If you want any help with the progress bars, send me a PM or email and I'll try to help.

There is always going to be a speed limitation with the script as Last.Fm does not allow more than one post per second from any IP address - consistently breaking this will result in your IP address being blocked and can lead to your account being blocked (if they can trace it back to an account). Does this script have anything in it to stop this from occuring?

[Psyker7]

Funny, I thought I remembered putting it in when I coded it, checked again and it's not there... will get to it tomorrow

Seemed to work fine for me though - Probably because the processing time of each xml page is longer than a second.

edit: Actually, I believe the 2.0 API no longer has this 1 second limit - as it needs an API key they know exactly what application is doing the requests, and it's likely I'll get notified if they get too much load on this key.
As the script shouldn't be run more than once a week, I doubt this will be an issue.

[Funkafonik]

That's great! Thanks for that script!

But like mentionned before, getting playcounts from the weekly charts isn't as precise as if we'd get them from our complete play history ( http://www.last.fm/user/USERNAME/tracks ). Mainly because weekly charts are static, once they are generated, they can't be changed, while the play history gets constantly updated by the user, some songs deleted, added, etc.

Would it be possible to have an option between getting playcounts from the weekly charts or our play history?

[nynaevelan]

Can you tell me where the log file is stored and what it is called because I cannot find it? And yes I turned the logging feature on.

Nyn

[Psyker7]

That's great! Thanks for that script!

But like mentionned before, getting playcounts from the weekly charts isn't as precise as if we'd get them from our complete play history ( http://www.last.fm/user/USERNAME/tracks ). Mainly because weekly charts are static, once they are generated, they can't be changed, while the play history gets constantly updated by the user, some songs deleted, added, etc.

Would it be possible to have an option between getting playcounts from the weekly charts or our play history?

The reason for not doing this are multiple. The main one being that page is not avAliable as an XML feed and so parsimg it would be much more difficult. Secondly I don't believe accuracy is nessesarily that important for this script.
There are always going to be some tracks missed no matter what I do - most notably if you have edited the track name or artist name since a song was scrobbled.
In my mind the idea here is not to keep the database perfectly in sync with last.fm but rather to fill in those large chunks of missing data from a portable device, another player or due to data loss which you could not sync playcounts with at all otherwise.

That all said I can take a look at it. No promises though.

[Psyker7]

Can you tell me where the log file is stored and what it is called because I cannot find it? And yes I turned the logging feature on.

Nyn

It *should* be MM\scripts\lastfmimport.vbs.log
I'll double check it's being created properly once I get home (can't really code vbs properly from an iPhone :p )

[nynaevelan]

Ok:

I didn't get one created, this is the line to change right"

Code: Select all

Const ForReading = 1, ForWriting = 2, ForAppending = 8, Logging = True, Timeout = 25

Nyn

[Funkafonik]

The reason for not doing this are multiple. The main one being that page is not avAliable as an XML feed and so parsimg it would be much more difficult. Secondly I don't believe accuracy is nessesarily that important for this script.
There are always going to be some tracks missed no matter what I do - most notably if you have edited the track name or artist name since a song was scrobbled.
In my mind the idea here is not to keep the database perfectly in sync with last.fm but rather to fill in those large chunks of missing data from a portable device, another player or due to data loss which you could not sync playcounts with at all otherwise.

That all said I can take a look at it. No promises though.

Fair enough, and I totally agree with you total accuracy isn't the most important thing, and to now have that option with your script is awesome!
Definitely one of the most useful script for us that have wanted to "recover/keep" playcounts forever!

If the parsing of our play history is possible, it'd only be a very nice finishing touch!

[Psyker7]

Ok:

I didn't get one created, this is the line to change right"

Code: Select all

Const ForReading = 1, ForWriting = 2, ForAppending = 8, Logging = True, Timeout = 25

Nyn

Correct.

Just checked myself - deleted the log file, and running the script did create it.
I did just realize there may have been issues with uploading not working correctly (the wrong version might have been uploaded >.< ) - might want to try downloading it again (clear browser cache first)

[Psyker7]

Update:
Changes: 1.2
- Status Bars!
- Code Tidy Up


The slowness isn't in the xml feeds, the slowness is in the database lookups which I'm sure can be optimized (nfi how myself atm)
I dump all the artists in a dictionary, each artist key points to a dictionary of tracknames, which contains playcounts.
I then iterate through this dictionary, doing a lookup on each trackname for a relevant artist: (code below is simplified obviously)

Code: Select all

	For Each ArtistName In ArtistsL.Keys
		For Each TrackTitle In ArtistsL.Item(ArtistName).Keys
			Dim list
			PlayCount = ArtistsL.Item(ArtistName).Item(TrackTitle)
			Set list = QueryLibrary (ArtistName,TrackTitle)
			If list.Count > 0 Then
				' Blah blah - update the playcounts if needed
			End If
			list.UpdateAll
		Next
	Next

Is there a better way? Say, pull all tracks by artist into some sort of datastructure, then search through / update that instead of querying the database for each track?

[Psyker7]

Changes: 1.3
- Database lookup optimizations (I hope!)
- Added a few more stats to the process
- Cleaned logging so enabling should only print essential stats about updated files

The database lookup is done per artist now, then the track list produced is compared to the list from last.fm for that artist. This appears to speed it up considerably, but I'm sure more could be done.

[Psyker7]

There is always going to be a speed limitation with the script as Last.Fm does not allow more than one post per second from any IP address - consistently breaking this will result in your IP address being blocked and can lead to your account being blocked (if they can trace it back to an account). Does this script have anything in it to stop this from occuring?

To further this:
Quoting from http://www.last.fm/api/tos

You will implement suitable caching in accordance with the HTTP headers sent with web service responses. You will not make more than 5 requests per originating IP address per second, averaged over a 5 minute period, without prior written consent. You agree to cache similar artist and any chart data (top tracks, top artists, top albums) for a minimum of one week.

With My 17 000 plays, and about a years worth of data, this script makes about 60 requests (about one request per week you've been a member of last.fm), generally in the space of 10 seconds (dependant on internet speeds of course)

That is only just over the 5 requests per second, but as it's only for 10 seconds you *could* run this script every minute and still be well under the limit.

Also, if it *is* exceeded, it will be *my* account that is blocked, not any other users, as the API key used in the script is linked directly to my account.

Good stuff to know I think

[trixmoto]

It might be worth adding the option for people to register for their own key and use it in your script, rather than provided your own - just a thought!

[pierre.osdoit]

-