From tony at clickcaster.com  Wed Jan  2 16:10:28 2008
From: tony at clickcaster.com (Tony Arcieri)
Date: Wed, 2 Jan 2008 14:10:28 -0700
Subject: [distribustream-talk] what algorithm?
In-Reply-To: <966599840801010848g77baddcfo5e9c8e07eaf5619@mail.gmail.com>
References: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
	<c7e6b2b00710311158m1a02980bwe566a6d87dd4b019@mail.gmail.com>
	<966599840711011209i65687895l4576278ca2fb1412@mail.gmail.com>
	<c7e6b2b00711011428mcfddb3bme584636ccbc3bcf2@mail.gmail.com>
	<966599840711060906q3560d14es173dd11a1b3b3000@mail.gmail.com>
	<c7e6b2b00711061112r5dd5cacg744f823f567a6d4f@mail.gmail.com>
	<966599840711061521t137c3154pc2be37e7c4990bf1@mail.gmail.com>
	<966599840711061524v856fe61t7952a35c85326db2@mail.gmail.com>
	<c7e6b2b00711061700t69481b08k4b6a33ea5df338fa@mail.gmail.com>
	<966599840801010848g77baddcfo5e9c8e07eaf5619@mail.gmail.com>
Message-ID: <c7e6b2b00801021310o7c53c2e7t95134f42b9be960@mail.gmail.com>

On Jan 1, 2008 9:48 AM, Roger Pack <rogerpack2005 at gmail.com> wrote:

> Might be a nice future project :)
>

I've seen something that does this now.  Zooko's Tahoe project (
allmydata.com) provides peer-to-peer backup and can stream data through a
decentralized peer network.


> In the finished product, how would that be accomplished, exactly?
>

pdtp:// URLs are the entry point (sort of like a torrent file).  You could
register your favorite program (something like Miro would be nice) with
pdtp:// URLs, then when you click on one it would be launched, the
peer-to-peer transfer initiated, and the video would start playing.


> concurrent meaning redundant?


No, simply meaning simultaneous.  The server picks the best candidate for
each distinct chunk based on the present state of the system.  There are no
redundant transfers initiated.

I suppose the ideal would be FEC
> encoding (well...maybe ideal maybe not--if it increases the number of
> bytes total that must be downloaded, when you could get away with just
> requesting the originals and get them just as fast, then you'd think
> it wouldn't help--I think the avalanche paper mostly showed that it
> helps with the last block problem in a good way).


I'd like to use erasure coding, but not on the chunks themselves.  Instead
it'd be nice for use in conjunction with a UDP chunk transfer protocol which
can utilize ICE/STUN for firewall traversal.

DistribuStream doesn't really suffer from a "last block problem" in that it
doesn't have to hammer the peer network looking for the harder to find
pieces.  Instead the server itself can pick up the slack for chunks which
are unavailable on the peer network.


> Another thought I had with my own stuff was the 'interlaced' block
> stuff--maybe one client gives you block y from beginning to end,
> another from end to beginning, when they meet then instruct them to
> close.  Or interlace them like client a sends you bytes 1, 3, 5, 7,
> then 2, 4, 6, whereas client b sends you bytes 2, 4, 6 then 1,3,5, 7
> (or 2,4,6,7,5,3,1--but the same point).  Then you might avoid needing
> FEC.


The problem with this approach in DistribuStream is that the server uses
reports of failed chunk transfers to calculate trust between peers.

In the case of a failed hash, the client reports a successful transfer along
with the hash it calculated and the server can then compare it against the
actual one, at which point it tells the client the hash didn't match.

If two peers are providing the same chunk simultaneously, there's no way to
determine if one was good and one was bad.  They'd both receive negative
credit.


> But I guess in reality this type of weirdness is only to avoid a slow
> last block problem--maybe it's premature optimization to think of such
> things.  Dunno :)
>

It's certainly not, however I believe this is mitigated by the protocol
design and implementation itself and additional measures are unnecessary.


> This project could work as semi-client-side-only if it just has a
> single person listening to a stream elect itself as the 'main' p2p
> streaming head or chair or whatever you call it, so I guess it kind of
> gets both sides.
>

The problem with the server delegating control of the network is the abuse
potential.  Clients are effectively a botnet and implicitly trust the
server.  A delegate authoritative peer could use the clients as a botnet to
do malicious things.

Perhaps the one way it could work:

The server can instruct all peers it wants to be managed by another that
they should open a connection to the delegate authority peer.  It'd also
contain a list of all peers presently connected to the network at that time.

The delegate can then regulate transfers among the peer whitelist.  However,
any additional peers that join afterward would still need to be regulated by
the central server.

There's a number of synchronization problems that could crop up in such a
scenario, however.

I think the best approach is still a cluster of trusted servers maintained
by a central authority.


> One good thing about having a central server would be that it makes an
> easy place for rendezvous among clients (not a lot of extra overhead
> for the same, like boot-strapping to a DHT first or what not), and
> also the server's around to coordinate if things go poorly for a
> client, and also could serve as a proxy for overcoming NAT's via STUN
> or what not.  So it makes streaming simple.
>

Yep, that's the goal: switch to a UDP chunk transfer protocol and use
ICE/STUN to traverse firewalls.


>
> Anyway good luck on it for now.  Wish me luck on my thesis!
>

Good luck!

-- 
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20080102/394503cb/attachment-0001.html 

From tony at clickcaster.com  Wed Jan  9 21:18:54 2008
From: tony at clickcaster.com (Tony Arcieri)
Date: Wed, 9 Jan 2008 19:18:54 -0700
Subject: [distribustream-talk] what algorithm?
In-Reply-To: <966599840801091811o78a7a622m571888f651aabcaf@mail.gmail.com>
References: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
	<c7e6b2b00711011428mcfddb3bme584636ccbc3bcf2@mail.gmail.com>
	<966599840711060906q3560d14es173dd11a1b3b3000@mail.gmail.com>
	<c7e6b2b00711061112r5dd5cacg744f823f567a6d4f@mail.gmail.com>
	<966599840711061521t137c3154pc2be37e7c4990bf1@mail.gmail.com>
	<966599840711061524v856fe61t7952a35c85326db2@mail.gmail.com>
	<c7e6b2b00711061700t69481b08k4b6a33ea5df338fa@mail.gmail.com>
	<966599840801010848g77baddcfo5e9c8e07eaf5619@mail.gmail.com>
	<c7e6b2b00801021310o7c53c2e7t95134f42b9be960@mail.gmail.com>
	<966599840801091811o78a7a622m571888f651aabcaf@mail.gmail.com>
Message-ID: <c7e6b2b00801091818j18c63309h3f5e7e05ea8ed908@mail.gmail.com>

On Jan 9, 2008 7:11 PM, Roger Pack <rogerpack2005 at gmail.com> wrote:

> That's an interesting idea--unreliable UDP + FEC => reliable UDP.
> Interesting :)
>

Yeah, it's certainly not an original one:

http://www.asperasoft.com/technology/solution/security.html

FASP claims a FEC/UDP-based data transfer protocol improves both latency and
throughput by eliminating all the ACKing TCP has to do.  One would suspect
this negatively effects congestion control...

-- 
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20080109/b9f2e8e4/attachment.html 

From tony at clickcaster.com  Wed Jan  9 21:19:32 2008
From: tony at clickcaster.com (Tony Arcieri)
Date: Wed, 9 Jan 2008 19:19:32 -0700
Subject: [distribustream-talk] what algorithm?
In-Reply-To: <c7e6b2b00801091818j18c63309h3f5e7e05ea8ed908@mail.gmail.com>
References: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
	<966599840711060906q3560d14es173dd11a1b3b3000@mail.gmail.com>
	<c7e6b2b00711061112r5dd5cacg744f823f567a6d4f@mail.gmail.com>
	<966599840711061521t137c3154pc2be37e7c4990bf1@mail.gmail.com>
	<966599840711061524v856fe61t7952a35c85326db2@mail.gmail.com>
	<c7e6b2b00711061700t69481b08k4b6a33ea5df338fa@mail.gmail.com>
	<966599840801010848g77baddcfo5e9c8e07eaf5619@mail.gmail.com>
	<c7e6b2b00801021310o7c53c2e7t95134f42b9be960@mail.gmail.com>
	<966599840801091811o78a7a622m571888f651aabcaf@mail.gmail.com>
	<c7e6b2b00801091818j18c63309h3f5e7e05ea8ed908@mail.gmail.com>
Message-ID: <c7e6b2b00801091819y74fb1ad3h9504ef054ef8b643@mail.gmail.com>

Whoops, here's the correct link:

http://www.asperasoft.com/technology/solution/speed.html

On Jan 9, 2008 7:18 PM, Tony Arcieri <tony at clickcaster.com> wrote:

> On Jan 9, 2008 7:11 PM, Roger Pack <rogerpack2005 at gmail.com> wrote:
>
> > That's an interesting idea--unreliable UDP + FEC => reliable UDP.
> > Interesting :)
> >
>
> Yeah, it's certainly not an original one:
>
> http://www.asperasoft.com/technology/solution/security.html
>
> FASP claims a FEC/UDP-based data transfer protocol improves both latency
> and throughput by eliminating all the ACKing TCP has to do.  One would
> suspect this negatively effects congestion control...
>
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com
>



-- 
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20080109/a33a7729/attachment.html