Last.fm Playcount Import [MM3]

[Psyker7]

Well here is a vbscript implimentation of a last.fm sync of playcounts / last played date.

It will find total playcounts of every track in your last.fm history, check if it is higher than the value in the database (for all tracks of that title/artist - even if multiples) and updates it if so.
It will also update the last played times for all tracks with a newer last played on last.fm than in your MM database.
===============
Useful if you lose your play counts, or you have a portable player (such as iPhone) which will scrobble tracks to last.fm OK, but won't update playcounts in MM (very annoying bug that)
It is *not* worth running more than once a week, as it uses your weekly track charts for totals, and these are only updated weekly.
===============

A log file is created automatically containing all tracks which were updated in a tab delimited file in your scripts directory.

A first run over a database can a few hours, but interrupting the script is safe, and running it later will pickup where it left off.
To speed it up somewhat, turn *OFF* autorate accurate, otherwise every track updated will need to be written too as the script processes.



Notes:

• Does *not* update MM's internal play histories. Feel free to use other scripts to fix this up (unlikely to be added as a feature here) viewtopic.php?f=2&t=31809
• If you have any issues... try version 2.0: Personally as much as I like having exact last played times, I'm more inclined to stick with the weekly summaries as it doesn't have the issue with the pagination while playing. This was never really intended to be an 'exact last played' script - rather just to catch some missing play counts.
  2.0 does this just fine

======

My first real scripting attempt for MM, thanks to:
Teknojnky - author of last.fmnode - for code for various functions, hacked a LOT.
Trixmoto - for mmip packaging of normal scripts, had me stumped for a while there

======================================================
Download:
http://psykar.com/scripts/LastFmImport.mmip
Previous versions:
http://psykar.com/scripts/
======================================================

I VERY STRONGLY recommend a database backup before you try this.

=====

' Changes: 3.0
' - Added support for last.fm's recent track history
'
' Changes: 2.0
' - Added support for updating last played times - these will be up to a week out though
'
' Changes: 1.12
' - Fix: More graceful xml checking, should catch ALL Invalid characters
'
' Changes: 1.11
' - Fix: More infalid xml characters checked
' - More graceful exits when errors occur
'
' Changes: 1.10
' - Fix: More invalid xml characters checked
'
' Changes: 1.9
' - Fix: Last.FM usernames not parsing correctly if containing special chars
'
' Changes: 1.8
' - Fix: Invalid ASCII characters stripped (hopefully - let me know if you find more!)
' Thanks to SinDenial and AndréVonDrei for testing!
' - More graceful error messages (for some, let me know if you get anything cryptic)
' - Check for invalid characters when writing update file - some seem to cause errors
' when the actual update went fine - needs improvment
'
' Changes: 1.7
' - Fix: Invalid apostrophes stripped, sadly this will make things less accurate
' but will reduce error messages for the moment
'
' Changes: 1.6
' - Fix: No longer case sensitive for track names
' - Fix: Error messages on timeouts are more helpful
'
' Changes: 1.5
' Better logging - by default a log file will be created listing tracks updated
' this file is LastFmImport.vbs.Updated.txt located in the scripts folder and is
' tab delimited.
' Better status bar messages as well I would like to think
'
' Changes: 1.4
' - HUGE speedup - no more .updateall() rather only update the track that needs it with
' updateDB()
'
' Changes: 1.3
' - Database lookup optimizations (I hope!)
' - Added a few more stats to the process
' - Cleaned logging so enabling should only print essential stats about updated files
'
' Changes: 1.2
' - Status Bars!
' - Code Tidy Up
'
' Changes: 1.1
' - Abstracted username
' - Pretty error messages
'
'ToDo:
'* Better UI
'* ???

[trixmoto]

Sounds like this will be a useful script for many people. I'm glad my MMIPs helped you with yours. If you want any help with the progress bars, send me a PM or email and I'll try to help.

There is always going to be a speed limitation with the script as Last.Fm does not allow more than one post per second from any IP address - consistently breaking this will result in your IP address being blocked and can lead to your account being blocked (if they can trace it back to an account). Does this script have anything in it to stop this from occuring?

[Psyker7]

Funny, I thought I remembered putting it in when I coded it, checked again and it's not there... will get to it tomorrow

Seemed to work fine for me though - Probably because the processing time of each xml page is longer than a second.

edit: Actually, I believe the 2.0 API no longer has this 1 second limit - as it needs an API key they know exactly what application is doing the requests, and it's likely I'll get notified if they get too much load on this key.
As the script shouldn't be run more than once a week, I doubt this will be an issue.

[Funkafonik]

That's great! Thanks for that script!

But like mentionned before, getting playcounts from the weekly charts isn't as precise as if we'd get them from our complete play history ( http://www.last.fm/user/USERNAME/tracks ). Mainly because weekly charts are static, once they are generated, they can't be changed, while the play history gets constantly updated by the user, some songs deleted, added, etc.

Would it be possible to have an option between getting playcounts from the weekly charts or our play history?

[nynaevelan]

Can you tell me where the log file is stored and what it is called because I cannot find it? And yes I turned the logging feature on.

Nyn

[Psyker7]

That's great! Thanks for that script!

But like mentionned before, getting playcounts from the weekly charts isn't as precise as if we'd get them from our complete play history ( http://www.last.fm/user/USERNAME/tracks ). Mainly because weekly charts are static, once they are generated, they can't be changed, while the play history gets constantly updated by the user, some songs deleted, added, etc.

Would it be possible to have an option between getting playcounts from the weekly charts or our play history?

The reason for not doing this are multiple. The main one being that page is not avAliable as an XML feed and so parsimg it would be much more difficult. Secondly I don't believe accuracy is nessesarily that important for this script.
There are always going to be some tracks missed no matter what I do - most notably if you have edited the track name or artist name since a song was scrobbled.
In my mind the idea here is not to keep the database perfectly in sync with last.fm but rather to fill in those large chunks of missing data from a portable device, another player or due to data loss which you could not sync playcounts with at all otherwise.

That all said I can take a look at it. No promises though.

[Psyker7]

Can you tell me where the log file is stored and what it is called because I cannot find it? And yes I turned the logging feature on.

Nyn

It *should* be MM\scripts\lastfmimport.vbs.log
I'll double check it's being created properly once I get home (can't really code vbs properly from an iPhone :p )

[nynaevelan]

Ok:

I didn't get one created, this is the line to change right"

Code: Select all

Const ForReading = 1, ForWriting = 2, ForAppending = 8, Logging = True, Timeout = 25

Nyn

[Funkafonik]

The reason for not doing this are multiple. The main one being that page is not avAliable as an XML feed and so parsimg it would be much more difficult. Secondly I don't believe accuracy is nessesarily that important for this script.
There are always going to be some tracks missed no matter what I do - most notably if you have edited the track name or artist name since a song was scrobbled.
In my mind the idea here is not to keep the database perfectly in sync with last.fm but rather to fill in those large chunks of missing data from a portable device, another player or due to data loss which you could not sync playcounts with at all otherwise.

That all said I can take a look at it. No promises though.

Fair enough, and I totally agree with you total accuracy isn't the most important thing, and to now have that option with your script is awesome!
Definitely one of the most useful script for us that have wanted to "recover/keep" playcounts forever!

If the parsing of our play history is possible, it'd only be a very nice finishing touch!

[Psyker7]

Ok:

I didn't get one created, this is the line to change right"

Code: Select all

Const ForReading = 1, ForWriting = 2, ForAppending = 8, Logging = True, Timeout = 25

Nyn

Correct.

Just checked myself - deleted the log file, and running the script did create it.
I did just realize there may have been issues with uploading not working correctly (the wrong version might have been uploaded >.< ) - might want to try downloading it again (clear browser cache first)

[Psyker7]

Update:
Changes: 1.2
- Status Bars!
- Code Tidy Up


The slowness isn't in the xml feeds, the slowness is in the database lookups which I'm sure can be optimized (nfi how myself atm)
I dump all the artists in a dictionary, each artist key points to a dictionary of tracknames, which contains playcounts.
I then iterate through this dictionary, doing a lookup on each trackname for a relevant artist: (code below is simplified obviously)

Code: Select all

   For Each ArtistName In ArtistsL.Keys
      For Each TrackTitle In ArtistsL.Item(ArtistName).Keys
         Dim list
         PlayCount = ArtistsL.Item(ArtistName).Item(TrackTitle)
         Set list = QueryLibrary (ArtistName,TrackTitle)
         If list.Count > 0 Then
            ' Blah blah - update the playcounts if needed
         End If
         list.UpdateAll
      Next
   Next

Is there a better way? Say, pull all tracks by artist into some sort of datastructure, then search through / update that instead of querying the database for each track?

[Psyker7]

Changes: 1.3
- Database lookup optimizations (I hope!)
- Added a few more stats to the process
- Cleaned logging so enabling should only print essential stats about updated files

The database lookup is done per artist now, then the track list produced is compared to the list from last.fm for that artist. This appears to speed it up considerably, but I'm sure more could be done.

[Psyker7]

There is always going to be a speed limitation with the script as Last.Fm does not allow more than one post per second from any IP address - consistently breaking this will result in your IP address being blocked and can lead to your account being blocked (if they can trace it back to an account). Does this script have anything in it to stop this from occuring?

To further this:
Quoting from http://www.last.fm/api/tos

You will implement suitable caching in accordance with the HTTP headers sent with web service responses. You will not make more than 5 requests per originating IP address per second, averaged over a 5 minute period, without prior written consent. You agree to cache similar artist and any chart data (top tracks, top artists, top albums) for a minimum of one week.

With My 17 000 plays, and about a years worth of data, this script makes about 60 requests (about one request per week you've been a member of last.fm), generally in the space of 10 seconds (dependant on internet speeds of course)

That is only just over the 5 requests per second, but as it's only for 10 seconds you *could* run this script every minute and still be well under the limit.

Also, if it *is* exceeded, it will be *my* account that is blocked, not any other users, as the API key used in the script is linked directly to my account.

Good stuff to know I think

[trixmoto]

It might be worth adding the option for people to register for their own key and use it in your script, rather than provided your own - just a thought!

[pierre.osdoit]

Hello,

I've been using iTunes for a while now, and because of growing discontent towards the app (and growing satisfaction using Mediamonkey), I am in the process of switching to MM full time. However, I've had a few difficulties to find ways to import some information I used a lot with iTunes, to make it up-to-date in MM, most notably the play count which I use much for smart playlists.

Hence my message: in order to get a farily up-to-date play count in MM, I've set up a Last.FM account, made it scrobble my iTunes library (which I've been able to "copy" to MM using m3u playlists), and I've then tried your script to get the information from Last.FM back to MM.

However, I get an error stating that "There is no charts for this user".
Going back to my last.fm account, it seems I actually have some playcount information up to date with iTunes, but no weekly charts..

Is there a way I could get the script to point to this address (http://www.lastfm.fr/user/posdoit/charts), which has the play history gathered from iTunes? Is there another way to import this information from iTunes?

Thanks a lot for your help for my newbie question!