How do I make a Django model from a tab-delimited data file?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How do I make a Django model from a tab-delimited data file?

Tom Tanner
I have a tab-delimited data file that looks something like this:


NAME S1903_C02_001E state county tract State-County-Tract-ID
Census Tract 201, Autauga County, Alabama 66000 01 001 020100 01001020100
Census Tract 202, Autauga County, Alabama 41107 01 001 020200 01001020200
Census Tract 203, Autauga County, Alabama 51250 01 001 020300 01001020300

I want to make a Django model named `MyModel` with three columns: "name", "data", and "geoid", which correspond to the file's columns "NAME", "S1903_C02_001E", and "State-County-Tract-ID." Can I do this via command line, or with a custom Python script? I'm running my Django project locally on a computer running Debian 9.3. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/389d8f38-6dbc-43a0-8a69-d20aa13ca844%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do I make a Django model from a tab-delimited data file?

Andréas Kühne
Hi,

You will have to parse the CSV file manually with a custom management command (at least that is what I would do). All you need to do is open the file, split each row with a "," and then  import the correct columns to the model.

You can also use something like pandas to convert the CSV file into something that you can create the models from. But that may be overkill in your case.

Regards,

Andréas

2018-01-09 3:38 GMT+01:00 Tom Tanner <[hidden email]>:
I have a tab-delimited data file that looks something like this:


NAME S1903_C02_001E state county tract State-County-Tract-ID
Census Tract 201, Autauga County, Alabama 66000 01 001 020100 01001020100
Census Tract 202, Autauga County, Alabama 41107 01 001 020200 01001020200
Census Tract 203, Autauga County, Alabama 51250 01 001 020300 01001020300

I want to make a Django model named `MyModel` with three columns: "name", "data", and "geoid", which correspond to the file's columns "NAME", "S1903_C02_001E", and "State-County-Tract-ID." Can I do this via command line, or with a custom Python script? I'm running my Django project locally on a computer running Debian 9.3. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/389d8f38-6dbc-43a0-8a69-d20aa13ca844%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK4qSCfv_uvC5R89ykNxHzd1yJU-GZMSCGjVrGeLudAPC%3Dfc-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do I make a Django model from a tab-delimited data file?

Kasper Laudrup
Hi,

On 2018-01-09 08:09, Andréas Kühne wrote:
> You will have to parse the CSV file manually with a custom management
> command (at least that is what I would do). All you need to do is open
> the file, split each row with a "," and then  import the correct columns
> to the model.

Unfortunately, CSV is a really bad, non-standardized format and simply
splitting each row with a "," will cause you all kinds of issues with
espacing values with a "," and other things.

I would suggest using the python CSV library for this:

https://docs.python.org/3/library/csv.html

Just though that was worth mentioning.

Kind regards,

Kasper Laudrup

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/83268fb2-22f6-2d6e-c14c-0015d77c48f0%40stacktrace.dk.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do I make a Django model from a tab-delimited data file?

Andréas Kühne
Kasper,

You are correct :-), I just assumed (probably incorrectly) that the data was just like his example - in that case it would have been easiest :-)

Regards,

Andréas

2018-01-09 9:27 GMT+01:00 Kasper Laudrup <[hidden email]>:
Hi,

On 2018-01-09 08:09, Andréas Kühne wrote:
You will have to parse the CSV file manually with a custom management command (at least that is what I would do). All you need to do is open the file, split each row with a "," and then  import the correct columns to the model.

Unfortunately, CSV is a really bad, non-standardized format and simply splitting each row with a "," will cause you all kinds of issues with espacing values with a "," and other things.

I would suggest using the python CSV library for this:

https://docs.python.org/3/library/csv.html

Just though that was worth mentioning.

Kind regards,

Kasper Laudrup

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/83268fb2-22f6-2d6e-c14c-0015d77c48f0%40stacktrace.dk.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK4qSCcFzovjvS8M4sgTd8euRcWKWCgqC2xJWcu_qjCvuqdpaw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do I make a Django model from a tab-delimited data file?

Tom Tanner
In reply to this post by Tom Tanner
Thanks you two. I'll check out that parser.

On Monday, January 8, 2018 at 9:38:44 PM UTC-5, Tom Tanner wrote:
I have a tab-delimited data file that looks something like this:


NAME S1903_C02_001E state county tract State-County-Tract-ID
Census Tract 201, Autauga County, Alabama 66000 01 001 020100 01001020100
Census Tract 202, Autauga County, Alabama 41107 01 001 020200 01001020200
Census Tract 203, Autauga County, Alabama 51250 01 001 020300 01001020300

I want to make a Django model named `MyModel` with three columns: "name", "data", and "geoid", which correspond to the file's columns "NAME", "S1903_C02_001E", and "State-County-Tract-ID." Can I do this via command line, or with a custom Python script? I'm running my Django project locally on a computer running Debian 9.3. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/3437a3bc-6a71-4a21-a410-7ede182cb83d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How do I make a Django model from a tab-delimited data file?

Scot Hacker-2
In reply to this post by Tom Tanner
Another approach is to use postgres' COPY command:

https://www.postgresql.org/docs/9.6/static/sql-copy.html

which knows how to parse and import tab-delimited files (and is crazy fast). Then, once the table exists, use Django's `inspectdb` management command to generate a Model corresponding to the table. It may need a little manual massaging afterwards, but I've done this on several projects and it works very well for most purposes.

./s


On Monday, January 8, 2018 at 6:38:44 PM UTC-8, Tom Tanner wrote:
I have a tab-delimited data file that looks something like this:


NAME S1903_C02_001E state county tract State-County-Tract-ID
Census Tract 201, Autauga County, Alabama 66000 01 001 020100 01001020100
Census Tract 202, Autauga County, Alabama 41107 01 001 020200 01001020200
Census Tract 203, Autauga County, Alabama 51250 01 001 020300 01001020300

I want to make a Django model named `MyModel` with three columns: "name", "data", and "geoid", which correspond to the file's columns "NAME", "S1903_C02_001E", and "State-County-Tract-ID." Can I do this via command line, or with a custom Python script? I'm running my Django project locally on a computer running Debian 9.3. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/708e0f1d-1cac-4321-a68e-697a4a8cd3da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.