Using SQL Maestro as a MM3 DB editor

Post a reply

Smilies
:D :) :( :o :-? 8) :lol: :x :P :oops: :cry: :evil: :roll: :wink:

BBCode is ON
[img] is ON
[url] is ON
Smilies are ON

Topic review
   

Expand view Topic review: Using SQL Maestro as a MM3 DB editor

by Sproaticus » Sat Jun 14, 2008 3:37 pm

MoDementia wrote:1. MM Devs need to include a separate libray for the way MM implemented IUNICODE collation that can be accessed by external database programs
2. Sqliteman provide an easy option to include/load such libraries when opening MM databases.
The less obvious choice - but my favorite which is why I keep repeating it - is to open up the Sqliteman API to scripting. Let the end-user decide how the custom collation is implemented. (In addition to a whole bunch of other major benefits.)

Javascript via Spidermonkey would be fine, but involves bit of work to embed into an app. Python 2.5 includes Sqlite support automatically, and (IMHO) is easier to integrate into a C app using SWIG (or even a C++ app using the Boost libs.)

by MoDementia » Fri Jun 13, 2008 12:48 am

petr.vanek wrote: sure. I know this sqlite API call. I just wanted to know if it's in some monkey's library or compiled into exe binary itself.

At last - I'm afraid I'm lost if it's in the final exe file - users cannot load it on demand in my app. I don't like to see this 3rd party collation hardcoded into Sqliteman so I'm searching for any "pluggable" solution.
I think I now have an understanding of the problem. I'm sorry that the issue is more MM related than Sqliteman. I was under the impression that Iunicode collation support was generic :( i.e. you turned it on or off.

I too am afraid that you are correct and that the information is in the final exe.

If thats the case then
1. MM Devs need to include a separate libray for the way MM implemented IUNICODE collation that can be accessed by external database programs
2. Sqliteman provide an easy option to include/load such libraries when opening MM databases.

by petr.vanek » Thu Jun 12, 2008 10:38 am

It's a sqlite API call - sqlite3_create_collation() and its cousins sqlite3_create_collation_v2() and sqlite3_create_collation16() - see http://www.sqlite.org/c3ref/create_collation.html for the docs.

It's also what I used in my Python example earlier in this thread. What would be really cool is if you could integrate SpiderMonkey (or whatever ECMAScript engine QT3 recommends), or whatever scripting language you prefer, and allow your end-users to do this sort of thing themselves. (A tall order, I admit! :wink:)
sure. I know this sqlite API call. I just wanted to know if it's in some monkey's library or compiled into exe binary itself.

At last - I'm afraid I'm lost if it's in the final exe file - users cannot load it on demand in my app. I don't like to see this 3rd party collation hardcoded into Sqliteman so I'm searching for any "pluggable" solution.

by Sproaticus » Wed Jun 11, 2008 12:13 pm

Anonymous wrote:
Bex wrote:I don't think it's stored in a separate dll. Can't you use this info?
jiri wrote:The way MM implemented IUNICODE collation could be easily described as WinAPI call:

Code: Select all

CompareStringW( 0x0409 /*US English*/, NORM_IGNORECASE | SORT_STRINGSORT, str1, len1/2, str2, len2/2) - 2;
Jiri
but how you forced it into sqlite db? You have to load it somehow with sqlite3 api commands, haven't you?
It's a sqlite API call - sqlite3_create_collation() and its cousins sqlite3_create_collation_v2() and sqlite3_create_collation16() - see http://www.sqlite.org/c3ref/create_collation.html for the docs.

It's also what I used in my Python example earlier in this thread. What would be really cool is if you could integrate SpiderMonkey (or whatever ECMAScript engine QT3 recommends), or whatever scripting language you prefer, and allow your end-users to do this sort of thing themselves. (A tall order, I admit! :wink:)

by Guest » Wed Jun 11, 2008 11:46 am

Bex wrote:I don't think it's stored in a separate dll. Can't you use this info?
jiri wrote:The way MM implemented IUNICODE collation could be easily described as WinAPI call:

Code: Select all

CompareStringW( 0x0409 /*US English*/, NORM_IGNORECASE | SORT_STRINGSORT, str1, len1/2, str2, len2/2) - 2;
Jiri
but how you forced it into sqlite db? You have to load it somehow with sqlite3 api commands, haven't you?

Re: IUNICODE collation implementation

by MoDementia » Wed Jun 11, 2008 7:44 am

petr.vanek wrote:hi all,

I'm solving this issue for Sqliteman (propably based on bugreport from somebody from MediaMonkey community):
***Shyly Puts Hand Up*** :P

by Bex » Wed Jun 11, 2008 7:00 am

I don't think it's stored in a separate dll. Can't you use this info?
jiri wrote:The way MM implemented IUNICODE collation could be easily described as WinAPI call:

Code: Select all

CompareStringW( 0x0409 /*US English*/, NORM_IGNORECASE | SORT_STRINGSORT, str1, len1/2, str2, len2/2) - 2;
Jiri

IUNICODE collation implementation

by petr.vanek » Wed Jun 11, 2008 6:36 am

hi all,

I'm solving this issue for Sqliteman (propably based on bugreport from somebody from MediaMonkey community):

http://sqliteman.com/bugtracker/view.php?id=134

Can you point me where is your IUNICODE collation stored, please? I mean in which DLL file?

by MDominik13 » Mon Jun 09, 2008 11:39 am

Guess I should have kept my mouth shut on this one... I have been using the Firefox add-on for the import and schema features (no queries). I've been using the MM SQL Script for any SELECT queries.

by Sproaticus » Sun Jun 08, 2008 8:29 pm

MDominik13 wrote:For any Firefox users, there's also an add-on that seems to work pretty well with the MM DB: SQLite Manager

I still prefer using the MM SQL script by Bex but the Firefox add-on does offer some nice features.
I like this add-on, and it's what I use on Linux where SQLiteSpy won't work well. But SQLite Manager has the same problem that SQLiteSpy has with the MM database, with the IUNICODE collation. How do you get around that using SQLite Manager?

Bex, I did see that script, and thank you. I've installed it now, but haven't played with it yet.

by MDominik13 » Sun Jun 08, 2008 5:32 pm

For any Firefox users, there's also an add-on that seems to work pretty well with the MM DB: SQLite Manager

I still prefer using the MM SQL script by Bex but the Firefox add-on does offer some nice features.

by Bex » Sun Jun 08, 2008 9:24 am

Well, COLLATE NOCASE is better since it doesn't care about the case. It is also only needed to apply it on TEXT fields (i.e COLLATE IUNICODE fields) and functions which is based on such fields. However, this doesn't make the SQL to always return expected result.
e.g. COLLATE NOCASE treats Unicode upper vs lower case characters as different characters.
Also, update/insert statements involving text fields is not possible in any external tool.


Btw, did you see the SQL-Viewer script?
http://www.mediamonkey.com/forum/viewtopic.php?t=24841
Perhaps it's not as fancy as the a "real" SQLite db editor but it doesn't have any of their drawbacks.

by Sproaticus » Sat Jun 07, 2008 12:04 pm

Replying to my own post...

You can bypass the IUNICODE collation using just plain SQL, making off-the-shelf SQLite tools like SQLiteSpy useful for MM hacking. There's no need to create a custom collation. The drawback is that collation-dependent queries, e.g. anything using an ORDER BY, will behave differently than MM. My guess is that this is largely restricted to issues involving sorting and grouping and whatnot, but I'm still pretty green with SQLite and probably wrong.

Anyway.

In order to bypass IUNICODE, you need to tag every expression in your SQL query with the COLLATE operator, e.g.:

Code: Select all

SELECT Songs.ID, Songs.SongTitle, Songs.Artist
FROM PodcastEpisodes, Songs
WHERE PodcastEpisodes.IDTrack = Songs.ID COLLATE BINARY
  AND Songs.Genre = 'Boring' COLLATE BINARY
ORDER BY Songs.ID;
The important bits there are the COLLATE BINARY after each expression.

I'm still using ORDER BY even though I've changed the collation - I'm making an assumption that the SYSTEM collation won't behave too differently than the IUNICODE collation when it comes to sorting numbers.

Possible solution for IUNICODE in Python 2.5

by Sproaticus » Mon Apr 21, 2008 7:50 pm

I haven't tested this against the behavior of MM's IUNICODE collation, but this seems to work in Python 2.5 using the sqlite3 module:

Code: Select all

import sqlite3

# define IUNICODE collation function
def iUnicodeCollate(s1, s2):
    return cmp(s1.lower(), s2.lower())

# connect to database
conn = sqlite3.connect('MM.DB')

# register our custom IUNICODE collation function
conn.create_collation('IUNICODE', iUnicodeCollate)

# run your query, business as usual
cursor = conn.execute('select * from Songs')
# ...etc...
The important bits are the definition of the iUnicodeCollate() function, and using conn.create_collation() to apply it to the db connection.

And like I said, I don't know if this behavior matches that of the one implemented in MediaMonkey. It does work without any errors, and behaves how I think it should behave, which is good enough for me. :)

If the Sqlite API in VB lets you define custom collations, this trick may work there as well. Unfortunately, I'm not well-versed in VB, and since my technique doesn't use ODBC I'm not even sure it'll translate to VB.

Some relevant links:

Sqlite docs on defining new collating sequences

Python API docs for Sqlite connection objects

Forum post by jiri in "sqlite db query error" which pointed me to the SQLite3_CreateCollation16() function.

by jiri » Thu Jan 31, 2008 6:49 am

The way MM implemented IUNICODE collation could be easily described as WinAPI call:

Code: Select all

CompareStringW( 0x0409 /*US English*/, NORM_IGNORECASE | SORT_STRINGSORT, str1, len1/2, str2, len2/2) - 2;
Jiri

Top