Python - Mysql Query Not Working

I am trying to load data into a MySQL table with the following Python script: conn = connect_db() cursor = conn.cursor() cursor.execute( 'LOAD DATA LOCAL INFILE ' + jobsummaryf

Solution 1:

It seems you need to add quotes for pipeline_name and grid

 cursor.execute(" ... pipeline='"+pipeline_name+"', grid='"+grid+"'")

Solution 2:

If you print out the constructed MySQL statement you'll see that you don't have the required single quotes around the string values in some of your assignments.

You will need to add single quotes around each of those values and, additionally, double-up any single-quotes in the string value (that is, galaxy should become 'galaxy' but Joe's Galaxy should become 'Joe''s Galaxy').

You can use a function along these lines:

 def qstr(in_string, quote_char="'"):

     return quote_char + in_string.replace(quote_char, quote_char*2) + quote_char

and then modify your code like this: "pipeline = " + qstr(pipeline_name).

Solution 3:

MySQLdb will deal with the quoting issue for you if you use parameters instead of all that string concatenation. The simplest way to do this is:

cursor.execute("SELECT * FROM tbl WHERE col1 = %s;", ('foo',))

%s is used as a placeholder for a parameter, and parameters are inserted in order; you must supply parameters for every placeholder in the query string. Alternatively, you can use named parameters with a dictionary mapping:

cursor.execute("SELECT * FROM tbl WHERE col1 = %(col1)s;", {'col1': 'foo'})

There is a limit to which parts of the statement can be replaced in this way (see the paramstyle attribute in the MySQLdb user guide):

Parameter placeholders can only be used to insert column values. They can not be used for other parts of SQL, such as table names, statements, etc.

So in your query, you can pass date, pipeline_name and grid as parameters, but not jobsummaryfile because it is a file path, not a column value. You can also use implicit string concatenation to make everything a little more readable, and simplify your statement by removing the unnecessary user variables:

query = ("LOAD DATA LOCAL INFILE " + jobsummaryfile + " INTO TABLE daily_job_summary "
         "FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' "
         "(jobname, queue, maphours, reducehours) "
         "set date=%s, pipeline=%s, grid=%s;")
curs.execute(query, (date, pipeline_name, grid))

The only reason you would need (@col1, @col2, @col3, @col4) is if you needed to manipulate all four columns of your input file before inserting them into your table. For instance, if you wanted to make sure your job name was always upper case, you could do it like this:

LOAD DATA LOCAL INFILE 'file.tsv'
    FIELDS TERMINATED BY '\t'
    LINES TERMINATED BY '\n'
    (@col1, queue, maphours, reducehours)
    SET jobname=UPPER(@col1), date='2014-01-05', pipeline='abcd', grid='AB';

But since all you're doing is inserting the data as-is, all those user variables just make the query longer and harder to read; leave them out.

---

Personally, I like to use Python's multi-line strings with my SQL statements. This lets me do two things: First, I can quickly copy/paste entire statements to and from Workbench for testing purposes; and second, I can start with a comment that will help me identify which queries are currently running when I'm checking the server administration tab. The down-side to this is that it's a little more verbose, especially when inserting things like table names.

For example:

load_from_file = """--load from {file}
LOAD DATA LOCAL INFILE {file}
    INTO TABLE daily_job_summary
    FIELDS TERMINATED BY '\t' 
    LINES TERMINATED BY '\n'
    (jobname, queue, maphours, reducehours)
    SET date=%s, pipeline=%s, grid=%s;
"""
curs.execute(query=load_from_file.format(file=jobsummaryfile), 
             args=(date, pipeline_name, grid))