Discussion:
[Arm-netbook] offlineimap syncing off of gmail...
Luke Kenneth Casson Leighton
2016-12-16 03:47:49 UTC
Permalink
after trying for 90 minutes to connect to accounts.google.com last
night, with over twenty refreshes, finally obtaining just the HTML of
a page then having to spend ANOTHER hour just to get the matching
CSS... only to run into HTTP proxy problems and having to clear the
cache and DO IT ALL OVER AGAIN....

... i'm absolutely sick and tired of the fucking fucked internet here,
how it's interfering with my ability to communicate, get source code
and much more.

so, i'm converting to offline imap, setting up the "prayer webmail"
client, a cyrus imap server, and syncing everything to my laptop now i
have the spare space on the 500GB SSD.

the only thing is... well... look at the numbers:

Folder INBOX [acc: lkcl]:
Copy message UID 143 (98/155030) Remote:INBOX -> Local
Folder INBOX.bak.sent [remote name [Gmail]/Sent Mail] [acc: lkcl]:
Copy message UID 307 (116/55244) Remote:[Gmail]/Sent Mail -> Local
Folder INBOX.bak.flagged [remote name [Gmail]/Starred] [acc: lkcl]:
Copy message UID 508 (161/3185) Remote:[Gmail]/Starred -> Local
Folder INBOX.bak.important [remote name [Gmail]/Important] [acc: lkcl]:
Copy message UID 296 (114/51381) Remote:[Gmail]/Important -> Local

that's nearly 10 GIGABYTES of mail over a ten year period (i created
the gmail account in 2006 when i was running out of space on my laptop
to hold all the email i had in mutt Maildirs).

55 *thousand* messages in the "sent mail" folder. a hundred and fifty
five thousand in the inbox.

after hacking offlineimap so that it would only use HTTPS proxy for
connecting to accounts.google.com but would connect direct to
imap.gmail.com (using a fixed IP address because the china government
MODIFIES the DNS entries to point to the wrong locations...)....

... i was able to initiate the download / sync process. connectivity
is so fucking slow it's about one message every... two to five
seconds. at this rate it's going to take TWO HUNDRED hours to sync
the entire mailbox. that's ten DAYS for god's sake.

we may actually have to go stay in a hotel in hong kong (where they
have a 100 mbytes / sec internet connection) just so i can sync all
the email.

dang.

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to
Julie Marchant
2016-12-16 04:02:03 UTC
Permalink
I'm pretty sure Gmail has an option to ignore emails before a certain
point for POP3. I don't know about IMAP, though.

Is there a particular reason you're using IMAP rather than POP? You
could very easily just download the new emails, leave them on the Gmail
server, and delete them from your computer when you're done with them at
the moment (with some custom settings on Gmail and/or on your email client).
--
Julie Marchant
https://onpon4.github.io

Protect your emails with GnuPG:
https://emailselfdefense.fsf.org
Luke Kenneth Casson Leighton
2016-12-16 04:11:41 UTC
Permalink
Post by Julie Marchant
I'm pretty sure Gmail has an option to ignore emails before a certain
point for POP3.
yeah i came across that when enabling imap last night
Post by Julie Marchant
I don't know about IMAP, though.
it doesn't
Post by Julie Marchant
Is there a particular reason you're using IMAP rather than POP?
from what i understand it's a far better protocol. POP3 if i recall
correctly, if you want to get even just the most recent messages you
have to get EVERYTHING. there's absolutely no practical way i can
grab 9.8gb of mail each time.

IMAP4 has a means to tell you which messages are most recent, it has
UIDs per message... it's basically designed for synchronisation.
Post by Julie Marchant
You
could very easily just download the new emails, leave them on the Gmail
server, and delete them from your computer when you're done with them at
the moment (with some custom settings on Gmail and/or on your email client).
i need access to the full ten years of history of the messages that
i've sent and received: it's proved invaluable numerous times. so
deleting is not an option. also, it's simply flat-out impractical to
consider at this point.

OfflineImapError: Server 'imap.gmail.com' closed connection, error on
SELECT '[Gmail]/Important'. Server said: command: EXAMINE => socket
error: <type 'exceptions.IOError'> - Too many read 0

... gotta love that... :)

so the combination of "need offline access" with "deleting
impractical and undesirable" basically means "use IMAP because it's
designed specifically for sychronisation.

offlineimap does *two way* synchronisation, so everything i write on
the laptop will be *uploaded* to gmail, so that i can continue to use
gmail in unusual circumstances.

later, once i'm confident with this setup, i can increase the hard
drive space on my server, replicate the synchronisation, and shut down
the gmail account (or turn it into a redirector).

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-ne
Christopher Havel
2016-12-16 04:30:01 UTC
Permalink
Hey Luke -- which is worse, in your opinion, Internet that sucks because of
lawful corporate greediness, or Internet that sucks because of governmental
meddling?

rant.

Here in my little town of Siler City, North Carolina, Murrika The Great
(yeah right) -- population roughly eight thousand according to the 2010
census (would've been probably ~9k but the chicken plant died Oct '09)...
we have basically three choices for Internet. Sort of. There's the phone
co, the cable TV co, and weird $#!* mobile/cell phone stuff, unless you
want to go to the local public library, which generously offers their
computers for one hour per week (three, if you bring your own computer and
leech their WiFi) -- mind you, they monitor and exclude certain content
there, and the chances of any real security being present on those machines
(and, therefore, a lack of a serious virus infestation, as well) is...
unlikely at best. (I'm being /phenomenally/ generously diplomatic here.)

Most of the town's populace is in "Greater Siler City", i.e. out in the
county (for you city dwellers, around here there's actual space between
small towns... the next one away is Pittsboro, and there's /sixteen miles/
of cow-patty countryside in between...) and that includes me. I'm about a
mile and a half past the Wal*Mart that's about a half-mile outside of the
old town limits (they extended 'em for Wally World, go figure).

The cable company, Charter Communications, will only really affordably
service you if you have a fairly short driveway -- if yours is 300ft
(~91.5m) or less, they will pay for the cable installation. If it's at
least an inch more, /you/ pay. We -- Mom and I -- were quoted, some years
ago, a price of us$3,059.95 (yes, three thousand fifty-nine dollars and
ninety-five cents) for that run. Five hundred feet. There's a technological
reason for that -- they use RG-6 coax for the signal, and 300ft is as far
as it can go without an expensive signal-booster in the line. We of course
declined the small fortune in expenditure -- we wouldn't have been able to
pay it even if we were that desperate -- and that was the end of that. Last
I heard, which was early this year, they weren't even servicing the road
I'm on any more at all. So that's option one.

Option two, the phone company (which, in addition to old-fashioned POTS
--plain old telephone service, i.e. landline phone-- offers DSL services)
is well known in these parts for defrauding customers by promising speeds
far, far in excess of what they can actually deliver, and then refusing to
be held responsible. There's option two.

Mom and I have option three. We have a Netgear router, model MBR1515, as
issued to --and locked to-- Verizon Wireless. Inside that router is a
little PCIe Mini Card 4g WWAN card, like you'd find in their (undoubtedly
horribly cheap and nasty, but I'm not fool enough to find out) tablet
offerings. We used to pay ~us$70 a month, but since we have a very very
grandfathered "unlimited" 4g plan -- i.e., we're able to evade their
incredibly strict data caps -- they've knocked our bill up us$20 every
billing cycle. Eventually, they'll probably try and force us to submit, at
which point, they'll be looking at losing a customer -- not that they
actually /care/, mind you.

Unfortunately, Mom and I are in their antenna shadow, so our "4g" service
is more like 2g service. You know those overpriced Dyson vacuum cleaners...
the ones that "never lose suction"...? Neither does Verizon. Our tenuous
tether is the modern equivalent of an acoustic coupler and a payphone with
a noisy line, and the part that rankles me even more than the corporate
apathy machine they have going, is that there's /absolutely no real need
for it/ -- they're basically extorting people because they /can/ -- sure,
they have radio-tower space to rent, and fiber and copper cables to lay,
and all that -- but they're literally choosing to shake down their
customers for the lunch money, because they know they can do it and get
away with it. There is absolutely no financial reason whatsoever that they
/have/ to set a data cap on people -- electrical pulses are by and large
free (the electric company sends us sixty of them every second for about
eleven cents every couple hours or thereabouts, although the rate depends
on the weather a bit), and they can /more/ than handle the infrastructure
costs they have, even with expansion. They're literally just being greedy
#^@%$ for the sake of being greedy #^@%$, and it pisses me off no end.

Oh, by the way, that router? It regularly overheats, even with a rather
gusty 120mm PC fan sitting in front of it... and consistently reboots
itself, spontaneously, at almost exactly 9:30pm every day... it's
maddening. Reboots take about five to ten minutes.

/rant.

So -- Luke -- what do you say...?
Luke Kenneth Casson Leighton
2016-12-16 04:50:53 UTC
Permalink
Post by Christopher Havel
Hey Luke -- which is worse, in your opinion, Internet that sucks because of
lawful corporate greediness, or Internet that sucks because of governmental
meddling?
all of the above and then technical incompetence as sugar on top
Post by Christopher Havel
on the weather a bit), and they can /more/ than handle the infrastructure
costs they have, even with expansion. They're literally just being greedy
So -- Luke -- what do you say...?
f***'em'all. fund 3G / LTE open hardware replacements (cell towers
*and* handsets), combined with 802.22 whitespace broadband operating
on between 140 to 700mhz for long-range communications, all as SDR...

... then get the equipment FCC Certified so that it's known to be safe...

... then completely ignore the FCC entirely except for actual genuine
safety reasons.

i've had it with hearing these kinds of stories [and experiencing them]

l.

p.s. yes, OpenBTS is developing 3G and 4G (LTE) as open hardware but
they forgot to develop a client at the same time.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachment
Christopher Havel
2016-12-16 04:54:43 UTC
Permalink
Unfortunately, government-owned and co-op Internet services are illegal in
NC. I can't really specify how that happened without getting into politics,
which I won't do, because that tends to cause electronic (and,
occasionally, IRL) fistfights. No thanks!
Luke Kenneth Casson Leighton
2016-12-16 05:14:12 UTC
Permalink
Post by Christopher Havel
Unfortunately, government-owned and co-op Internet services are illegal in
NC. I can't really specify how that happened without getting into politics,
which I won't do, because that tends to cause electronic (and,
occasionally, IRL) fistfights. No thanks!
look up robert david steele.

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-***@files.phcom
Christopher Havel
2016-12-16 05:24:02 UTC
Permalink
Looked him up on Wikipedia. Sounds to me like he's a bit of a wingnut,
sorry. I'm sure he's done good things for the open source cause -- but
anyone who uses the term "false flag" about... well, let's just leave it at
'well-known events', is of questionable sanity at best, in my book.
/Triply/ so for anyone who's been a guest of that Alex Jones guy. Sure, I
believe in aliens -- I'll admit that -- but for me it's almost entirely a
matter of statistics. I /certainly/ don't think there are lizard people
from Mars taking over (or at least attempting to) multiple major national
governments, FFS... Alex Jones does.
Luke Kenneth Casson Leighton
2016-12-16 05:52:05 UTC
Permalink
Post by Christopher Havel
Looked him up on Wikipedia. Sounds to me like he's a bit of a wingnut,
don't for god's sake trust what you read on wikipedia, it's
"ignorance by consensus".

google the 2014 guardian newspaper article.

he's HIGHLY respected in the intelligence community.

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to
Christopher Havel
2016-12-16 06:08:17 UTC
Permalink
I'm afraid I'm one of those ignorant sheeple* who has an unshakable trust
in Wiki. Sorry. I was raised to value sanity and common sense... I know
you've had a bad experience with Wiki -- you've mentioned that, once or
twice -- but I've yet to find anything on there that doesn't fit the
definition of trustworthy that's in my dictionary. I should probably
mention that I also don't believe in grudges -- I've learned that the hard
way -- it just makes one sound old, bitter, and cratchety well beyond one's
actual age. Not to mention it's bad for one's blood pressure and mental
health in general.

If you're going to insist on going all conspiracy theorist on me, I'm
afraid it'll fall on some rather deaf ears. Again, sorry, but, in addition
to the above, Alex Jones and his ilk are the best reason I've ever heard
that 'manure salesman' should be a licensed profession. My head's solidly
grounded, and, thank you just the same, but I'd like to keep it that way...

I think, actually, I'll take this opportunity to ring off for the night.
It's ~1am here, and that puts me a solid hour overdue...

* http://xkcd.com/1013/ -- well, maybe not... ;)

On Fri, Dec 16, 2016 at 12:52 AM, Luke Kenneth Casson Leighton <
Post by Luke Kenneth Casson Leighton
Post by Christopher Havel
Looked him up on Wikipedia. Sounds to me like he's a bit of a wingnut,
don't for god's sake trust what you read on wikipedia, it's
"ignorance by consensus".
google the 2014 guardian newspaper article.
he's HIGHLY respected in the intelligence community.
l.
_______________________________________________
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Luke Kenneth Casson Leighton
2016-12-16 07:51:16 UTC
Permalink
Post by Christopher Havel
I'm afraid I'm one of those ignorant sheeple* who has an unshakable trust
in Wiki. Sorry. I was raised to value sanity and common sense... I know
you've had a bad experience with Wiki -- you've mentioned that, once or
twice --
it was four separate pages, four completely separate bunches of
fuckwits, over something like an eight year period.
Post by Christopher Havel
If you're going to insist on going all conspiracy theorist on me, I'm
read the guardian article. i can't be bothered with conspiracy
theories, they're a waste of my time.

there's a conversation i'd like to have, which i cannot do if you're
going to dismiss out-of-hand one of the key strategic people because
of some fuckwits on wikipedia.

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-***@file
Luke Kenneth Casson Leighton
2016-12-16 14:48:43 UTC
Permalink
heh this guy. i was read that article the other day :) and found some
more on shared website, inc ones written by a curtain someone from this
list, hint hint ;)
... so you know where i'm going with this. it was the "open
manifesto" (open health, open intelligence, open source, open
information etc.) that robert steele wrote, after being accidentally
inspired by the open approach in intelligence and finding that people
in the wider intelligence community *liked* it, that caught my eye.
https://www.theguardian.com/environment/earth-insight/2014/jun/19/open-source-revolution-conquer-one-percent-cia-spy
that's the one.
basically it looks like hes a open minded, innovator type person, who
questioned the current way and attitude of the way the security services
do things and looked, thought, seeked, tried,exploded alt ways
you mean explored not exploded :)

yep, after forming the u.s. navy seals intelligence corp, he looked
at setting up a conference on "open sources in intelligence", and it
was highly successful. but his bosses at the CIA didn't like it and
refused to fund the next conference or permit him to go to it... so he
quit.

basically he can see that the current approach is self-perpetuating,
corrupt, and, from being *in* the CIA he *KNOWS* damn well that their
approach is completely ineffective. aaaalll that
information-gathering, with the associated violations of privacy, and
it's completely ineffective at achieving its intended goal: stopping
"terrorists" and criminals alike because it *doesn't provide
actionable intelligence*.

it's *word-of-mouth* and sources *outside* of the intelligence
community that *actually* provides the valuable, actionable
intelligence and corroborated facts!

let me reiterate: people in the U.S. govt do *NOT* go to the CIA to
find sources of information on which to make intelligence decisions,
they go out on... guess what? the internet! and they communicate
with people via... the internet!

so it's really not hard to imagine why robert made the intuitive leap
that security is achieved through open access to information, but
that's not enough: he had to witness how utterly ineffective the CIA
really was (is), from the inside.
the article still holds out on its own anyway. still makes for new
incites to alt way of security being done, one which sounds like a far
less bad will spreading one.
yeah.

so, part of the series i'm writing is to advocate the funding of
OpenBTS and projects like it, and to curtail the powers of the FCC.

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-***@files.phcomp.c

Julie Marchant
2016-12-16 14:11:14 UTC
Permalink
Post by Luke Kenneth Casson Leighton
don't for god's sake trust what you read on wikipedia, it's
"ignorance by consensus".
google the 2014 guardian newspaper article.
he's HIGHLY respected in the intelligence community.
You mean this one?

https://www.theguardian.com/environment/earth-insight/2014/jun/19/open-source-revolution-conquer-one-percent-cia-spy

It's written by Nafeez Ahmed, a conspiracy theorist who thinks 9/11 was
an inside job. That raises my skepticism alarm right away. Just trusting
whatever some conspiracy theorist wrote in an article on the Guardian
would be foolish. That said, I don't even know what this article has to
do with Internet services or why you brought it up. It starts by
promoting a bunch of books these two wrote and then goes on to talk
about "open source intelligence", the idea of intelligence agencies that
don't collect any secret data. Nothing in there at all is about Internet
services.

The claim that Steele is "hugely respected" in the article, the only
part of it that you specifically referenced, is entirely unsourced. It
seems to just be Ahmed's unsubstantiated opinion.
--
Julie Marchant
https://onpon4.github.io

Protect your emails with GnuPG:
https://emailselfdefense.fsf.org
Philip Hands
2016-12-16 08:35:22 UTC
Permalink
Post by Luke Kenneth Casson Leighton
after trying for 90 minutes to connect to accounts.google.com last
night, with over twenty refreshes, finally obtaining just the HTML of
a page then having to spend ANOTHER hour just to get the matching
CSS... only to run into HTTP proxy problems and having to clear the
cache and DO IT ALL OVER AGAIN....
... i'm absolutely sick and tired of the fucking fucked internet here,
how it's interfering with my ability to communicate, get source code
and much more.
so, i'm converting to offline imap, setting up the "prayer webmail"
client, a cyrus imap server, and syncing everything to my laptop now i
have the spare space on the 500GB SSD.
If you can deal with emacs, have a look at notmuch

Even if you cannot -- you should probably have a look at notmuch
(there's a vim-based thingumy, and there are other front ends too)
Post by Luke Kenneth Casson Leighton
Copy message UID 143 (98/155030) Remote:INBOX -> Local
Copy message UID 307 (116/55244) Remote:[Gmail]/Sent Mail -> Local
Copy message UID 508 (161/3185) Remote:[Gmail]/Starred -> Local
Copy message UID 296 (114/51381) Remote:[Gmail]/Important -> Local
that's nearly 10 GIGABYTES of mail over a ten year period (i created
the gmail account in 2006 when i was running out of space on my laptop
to hold all the email i had in mutt Maildirs).
55 *thousand* messages in the "sent mail" folder. a hundred and fifty
five thousand in the inbox.
after hacking offlineimap so that it would only use HTTPS proxy for
connecting to accounts.google.com but would connect direct to
imap.gmail.com (using a fixed IP address because the china government
MODIFIES the DNS entries to point to the wrong locations...)....
... i was able to initiate the download / sync process. connectivity
is so fucking slow it's about one message every... two to five
seconds. at this rate it's going to take TWO HUNDRED hours to sync
the entire mailbox. that's ten DAYS for god's sake.
we may actually have to go stay in a hotel in hong kong (where they
have a 100 mbytes / sec internet connection) just so i can sync all
the email.
If you point notmuch at that it'll say something like:

523,567 mails, that's not much mail...

and you'll be able to do similar searchy things to what you probably got
used to with google.

I tend to shunt my inbox into an archive directory so that I can use
dumb imap on my phone and not need to fill the phone with 110k of mails,
but I know I've got more mail than you're talking about, and searches
return useful results in sub-second mostly.

% find ./Maildir -type f | wc -l
559007
% du -hs Maildir
12G Maildir

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Luke Kenneth Casson Leighton
2016-12-16 09:20:23 UTC
Permalink
Post by Philip Hands
Post by Luke Kenneth Casson Leighton
after trying for 90 minutes to connect to accounts.google.com last
night, with over twenty refreshes, finally obtaining just the HTML of
a page then having to spend ANOTHER hour just to get the matching
CSS... only to run into HTTP proxy problems and having to clear the
cache and DO IT ALL OVER AGAIN....
... i'm absolutely sick and tired of the fucking fucked internet here,
how it's interfering with my ability to communicate, get source code
and much more.
so, i'm converting to offline imap, setting up the "prayer webmail"
client, a cyrus imap server, and syncing everything to my laptop now i
have the spare space on the 500GB SSD.
If you can deal with emacs, have a look at notmuch
i remember you mentioning it a few years ago, so yes, i installed it
in preparation, a few days ago. only did cyrus last night so haven't
got round to configuring it
Post by Philip Hands
Even if you cannot -- you should probably have a look at notmuch
(there's a vim-based thingumy, and there are other front ends too)
yeah i decided to go with the web-based front-end, it seems sane and
like prayer-webmail-imap-client doesn't depend on php...
Post by Philip Hands
Post by Luke Kenneth Casson Leighton
we may actually have to go stay in a hotel in hong kong (where they
have a 100 mbytes / sec internet connection) just so i can sync all
the email.
523,567 mails, that's not much mail...
:)
Post by Philip Hands
and you'll be able to do similar searchy things to what you probably got
used to with google.
goooood
Post by Philip Hands
I tend to shunt my inbox into an archive directory so that I can use
dumb imap on my phone and not need to fill the phone with 110k of mails,
yehhh the whole reason i started using gmail was because i'd had HDDs
die, i had several gig of maildirs, not enough space: when i started
with SSDs the capacity was just far too small. now it's up to sane
amounts.
Post by Philip Hands
but I know I've got more mail than you're talking about, and searches
return useful results in sub-second mostly.
faantastic. yeah i can't remember what the dependency was for the
indexing but i've seen it used before and it's pretty damn good.
Post by Philip Hands
% find ./Maildir -type f | wc -l
559007
% du -hs Maildir
12G Maildir
frickin mad, innit? :)

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-
Loading...