<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Avi :) &#187; perl 5.10</title>
	<atom:link href="http://blog.avi.co/wp/tag/perl5-10/feed/" rel="self" type="application/rss+xml" />
	<link>http://blog.avi.co</link>
	<description>Open Source Software Hippy and all-round pedant</description>
	<lastBuildDate>Mon, 07 Jun 2010 14:34:05 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9.1</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Splitting massive MySQL dumps</title>
		<link>http://blog.avi.co/wp/splitting-massive-mysql-dumps/</link>
		<comments>http://blog.avi.co/wp/splitting-massive-mysql-dumps/#comments</comments>
		<pubDate>Wed, 13 Jan 2010 23:45:22 +0000</pubDate>
		<dc:creator>Avi</dc:creator>
				<category><![CDATA[Geekery]]></category>
		<category><![CDATA[code]]></category>
		<category><![CDATA[dump]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[perl]]></category>
		<category><![CDATA[perl 5.10]]></category>

		<guid isPermaLink="false">http://aviswebsite.co.uk/?p=510</guid>
		<description><![CDATA[As I posted yesterday, I have a massive MySQL dump to import. I tried BigDump, but one of the tables kept producing errors and so BigDump would exit. I don't need the whole db imported, so I wrote this to split it by table. It produces a new sql file for every table it finds, [...]]]></description>
			<content:encoded><![CDATA[<p>As I <a href="http://aviswebsite.co.uk/wp/massive-dumps-with-mysql/">posted yesterday</a>, I have a massive MySQL dump to import. I tried BigDump, but one of the tables kept producing errors and so BigDump would exit. I don't need the whole db imported, so I wrote this to split it by table. It produces a new sql file for every table it finds, numbered sequentially so if you process them in alphabetical order it's the equivalent of the whole dump. <tt>USE</tt> statements get their own files in the same sequence.</p>
<pre class="perl"><span style="color: #808080; font-style: italic;">#! /usr/bin/perl</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">use</span> strict;
<span style="color: #000000; font-weight: bold;">use</span> warnings;
<span style="color: #000000; font-weight: bold;">use</span> <span style="color: #cc66cc;">5.010</span>;
&nbsp;
<span style="color: #b1b100;">my</span> <span style="color: #0000ff;">$dump_file</span> = <span style="color: #0000ff;">$ARGV</span><span style="color: #66cc66;">&#91;</span><span style="color: #cc66cc;">0</span><span style="color: #66cc66;">&#93;</span>;
&amp;usage<span style="color: #66cc66;">&#40;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #b1b100;">if</span> !<span style="color: #0000ff;">$dump_file</span>;
&nbsp;
say <span style="color: #ff0000;">&quot;using &quot;</span>.<span style="color: #0000ff;">$dump_file</span>;
&nbsp;
<span style="color: #b1b100;">my</span> <span style="color: #66cc66;">&#40;</span><span style="color: #0000ff;">$line</span>, <span style="color: #0000ff;">$table</span>,<span style="color: #0000ff;">@query</span>, <span style="color: #0000ff;">$file_number</span>,<span style="color: #0000ff;">$file_name</span><span style="color: #66cc66;">&#41;</span>;
<span style="color: #b1b100;">my</span> <span style="color: #0000ff;">$line_number</span> = <span style="color: #cc66cc;">1</span>;
<span style="color: #b1b100;">my</span> <span style="color: #0000ff;">$find_count</span> = <span style="color: #cc66cc;">0</span>;
&nbsp;
<a href="http://perldoc.perl.org/functions/open.html"><span style="color: #000066;">open</span></a><span style="color: #66cc66;">&#40;</span>DUMP_IN, <span style="color: #ff0000;">&quot;&lt; $dump_file&quot;</span><span style="color: #66cc66;">&#41;</span>;
        <span style="color: #b1b100;">while</span><span style="color: #66cc66;">&#40;</span><span style="color: #009999;">&lt;DUMP_IN&gt;</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#123;</span>
                <span style="color: #b1b100;">my</span> <span style="color: #0000ff;">$line</span> = <span style="color: #0000ff;">$_</span>;
                <span style="color: #b1b100;">if</span> <span style="color: #66cc66;">&#40;</span>/^USE\<a href="http://perldoc.perl.org/functions/s.html"><span style="color: #000066;">s</span></a>.<span style="color: #66cc66;">&#40;</span>\w+<span style="color: #66cc66;">&#41;</span>./<span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#123;</span>
                        say <span style="color: #ff0000;">&quot;changing db: &quot;</span>.$<span style="color: #cc66cc;">1</span>;
                        <span style="color: #0000ff;">$file_name</span> = &amp;make_file_name<span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">&quot;USE_$1&quot;</span>, <span style="color: #ff0000;">&quot;$find_count&quot;</span><span style="color: #66cc66;">&#41;</span>;
                        &amp;write_USE<span style="color: #66cc66;">&#40;</span><span style="color: #0000ff;">$file_name</span>, <span style="color: #0000ff;">$line</span><span style="color: #66cc66;">&#41;</span>;
                        <span style="color: #0000ff;">$find_count</span>++;
                <span style="color: #66cc66;">&#125;</span><span style="color: #b1b100;">elsif</span> <span style="color: #66cc66;">&#40;</span>/^-- Table structure <span style="color: #b1b100;">for</span> table .<span style="color: #66cc66;">&#40;</span>.+<span style="color: #66cc66;">&#41;</span>./<span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#123;</span>
			<span style="color: #808080; font-style: italic;">## If the current line is the beginning of a table definition</span>
			<span style="color: #808080; font-style: italic;">## and @query is defined, then @query must be full of the previous</span>
			<span style="color: #808080; font-style: italic;">## table, so we want to process it now:</span>
                        <span style="color: #b1b100;">if</span> <span style="color: #66cc66;">&#40;</span><span style="color: #0000ff;">@query</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#123;</span>
                        <span style="color: #0000ff;">$file_name</span> = &amp;make_file_name<span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">&quot;$table&quot;</span>, <span style="color: #ff0000;">&quot;$find_count&quot;</span><span style="color: #66cc66;">&#41;</span>;
                                <a href="http://perldoc.perl.org/functions/open.html"><span style="color: #000066;">open</span></a><span style="color: #66cc66;">&#40;</span>OUTPUT, <span style="color: #ff0000;">&quot;&gt;$file_name&quot;</span><span style="color: #66cc66;">&#41;</span>;
                                        <span style="color: #b1b100;">foreach</span><span style="color: #66cc66;">&#40;</span><span style="color: #0000ff;">@query</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#123;</span>
                                                <a href="http://perldoc.perl.org/functions/print.html"><span style="color: #000066;">print</span></a> OUTPUT <span style="color: #0000ff;">$_</span>;
                                        <span style="color: #66cc66;">&#125;</span>
                                <a href="http://perldoc.perl.org/functions/close.html"><span style="color: #000066;">close</span></a> OUTPUT;
                                <a href="http://perldoc.perl.org/functions/undef.html"><span style="color: #000066;">undef</span></a> <span style="color: #0000ff;">@query</span>;
                        <span style="color: #66cc66;">&#125;</span>
                        <span style="color: #0000ff;">$table</span> = $<span style="color: #cc66cc;">1</span>;
                        <span style="color: #0000ff;">$find_count</span>++;
                <span style="color: #66cc66;">&#125;</span>
                <span style="color: #b1b100;">next</span> <span style="color: #b1b100;">unless</span> <span style="color: #0000ff;">$table</span>;
                <a href="http://perldoc.perl.org/functions/push.html"><span style="color: #000066;">push</span></a> <span style="color: #0000ff;">@query</span>, <span style="color: #0000ff;">$line</span>;
&nbsp;
                <span style="color: #0000ff;">$line_number</span>++;
        <span style="color: #66cc66;">&#125;</span>
<a href="http://perldoc.perl.org/functions/close.html"><span style="color: #000066;">close</span></a> DUMP_IN;
say <span style="color: #0000ff;">$line_number</span>;
&nbsp;
<span style="color: #808080; font-style: italic;">## Subroutines!</span>
<span style="color: #000000; font-weight: bold;">sub</span> write_USE<span style="color: #66cc66;">&#40;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #66cc66;">&#123;</span>
        <span style="color: #b1b100;">my</span><span style="color: #66cc66;">&#40;</span><span style="color: #0000ff;">$filename</span>, <span style="color: #0000ff;">$line</span><span style="color: #66cc66;">&#41;</span> = <span style="color: #0000ff;">@_</span><span style="color: #66cc66;">&#91;</span><span style="color: #cc66cc;">0</span>,<span style="color: #cc66cc;">1</span><span style="color: #66cc66;">&#93;</span>;
        <a href="http://perldoc.perl.org/functions/open.html"><span style="color: #000066;">open</span></a> <span style="color: #66cc66;">&#40;</span>OUTPUT, <span style="color: #ff0000;">&quot;&gt;$filename&quot;</span><span style="color: #66cc66;">&#41;</span>;
        <a href="http://perldoc.perl.org/functions/print.html"><span style="color: #000066;">print</span></a> OUTPUT <span style="color: #0000ff;">$line</span>;
        <a href="http://perldoc.perl.org/functions/close.html"><span style="color: #000066;">close</span></a> OUTPUT;
<span style="color: #66cc66;">&#125;</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">sub</span> make_file_name<span style="color: #66cc66;">&#40;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #66cc66;">&#123;</span>
        <span style="color: #b1b100;">my</span> <span style="color: #66cc66;">&#40;</span><span style="color: #0000ff;">$type</span>, <span style="color: #0000ff;">$number</span><span style="color: #66cc66;">&#41;</span> = <span style="color: #0000ff;">@_</span><span style="color: #66cc66;">&#91;</span><span style="color: #cc66cc;">0</span>,<span style="color: #cc66cc;">1</span><span style="color: #66cc66;">&#93;</span>;
        <span style="color: #0000ff;">$number</span> = <a href="http://perldoc.perl.org/functions/sprintf.html"><span style="color: #000066;">sprintf</span></a><span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">&quot;%05d&quot;</span>, <span style="color: #0000ff;">$number</span><span style="color: #66cc66;">&#41;</span>;
        <span style="color: #0000ff;">$file_name</span>=<span style="color: #0000ff;">$number</span>.<span style="color: #ff0000;">&quot;_&quot;</span>.<span style="color: #0000ff;">$type</span>.<span style="color: #ff0000;">&quot;.sql&quot;</span>;
        <a href="http://perldoc.perl.org/functions/return.html"><span style="color: #000066;">return</span></a> <span style="color: #0000ff;">$file_name</span>;
<span style="color: #66cc66;">&#125;</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">sub</span> usage<span style="color: #66cc66;">&#40;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #66cc66;">&#123;</span>
        say <span style="color: #ff0000;">&quot;Error: missing arguments.&quot;</span>;
	say <span style="color: #ff0000;">&quot;Usage:&quot;</span>;
	say <span style="color: #ff0000;">&quot;$0 [MYSQL_DUMP]&quot;</span>;
        <a href="http://perldoc.perl.org/functions/exit.html"><span style="color: #000066;">exit</span></a> <span style="color: #cc66cc;">1</span>;
<span style="color: #66cc66;">&#125;</span>
&nbsp;</pre>
<p>A small downside is that this replaces my 2.5Gb file with about 1800 smaller ones. A scripted importer is to follow.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.avi.co/wp/splitting-massive-mysql-dumps/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>
