Is there a better ETL approach? Data Warehouse is all about loading data from source to target. Traditionally, source system produces fil...

My answer to a  curious ETL question

Answer by William Sharp:

Connect directly and schedule the etl jobs to run at a rate that matches the update frequency of the source.  you can profile the source update dates to figure that out.

View Answer on Quora

What aspects of Data Quality influence the impact on Business Value?

Answer by William Sharp:

when I read your question the one word that stood out to me was documents.  to me, this means data entry deviations. In the context of the 6dimensions of data quality this means consistency checks (is a value referred to/spelled the same way across all the documents), completeness checks (is all the required information present in every document), accuracy checks (is each value from ever document correctly represented), duplication checks (is the information in the documents redundant)

View Answer on Quora

Why should I use an existing ETL vs writing my own in Python for my data warehouse needs?

Answer by William Sharp:

Unless you want to enter the ETL software market, you will spend a lot of time writing software instead of migrating / managing data (which I am assuming is your real job). If this tool is going to be used by many developers in an enterprise setting, you will spend a lot of time writing code to manage the ETL code. Probably more than you think

This is classic buy over build ... buy because there are people dedicated to solving issues you haven't even thought of yet. Build if your needs are so unique that there isn't a solution out there (extremely rare)

View Answer on Quora

What things does a data scientist do?

Answer by William Sharp:

Put succinctly it is data management and statistics.  collect, clean and migrate data (just like everybody else does) and perform statistics on that data.  data science is a new marketing term that goes along with big data.  I'm not down playing statistics. I love stats. the 'science' part is a little (a lot) pretentious imho

View Answer on Quora

What are some good resources for learning about distributed computing? Why?

Answer by Alex Kamil:

Prerequisites

• Unix shell basics: http://www.amazon.com/Unix-Progr...
• C: http://www.amazon.com/Programmin...
• OS basics: http://www.amazon.com/Andrew-S-T... , http://www.amazon.com/Linux-Kern...
• Unix Programming: http://www.kohala.com/start/
• Networking Basics: http://www.amazon.com/Computer-N... , more advanced: http://web.mit.edu/dimitrib/www/...
  
• Sockets: http://www.amazon.com/TCP-Socket... , http://www.amazon.com/Foundation... , and network programming: http://www.kohala.com/start/

• RPC: Remote Procedure Calls (RPC)
• XMPP: http://web.sarathlakshman.com/Ar...

• Transmission of information: http://www.amazon.com/Informatio... , http://www.inference.phy.cam.ac.... and network coding
• Intro to concurrency: http://www.amazon.com/Practical-...
• Java: What questions are Java Software Engineers seeing the most of on technical interviews?
• Data structures and algorithms: Learning Algorithms: What are the most learner-friendly resources for learning about algorithms?

Courses

• CS525: http://www.cs.uiuc.edu/class/sp1...
• 6.824: http://pdos.csail.mit.edu/6.824/...
• 6.828: http://pdos.csail.mit.edu/6.828/...
• CS264: http://www.cs264.org/lectures/le...
• CS294: http://www.cs.berkeley.edu/~oded...
• CS 707: http://www.cs.gmu.edu/~setia/cs7...
• Advanced Computer Science Courses: http://the-paper-trail.org/blog/...
• CS696: http://www.eli.sdsu.edu/courses/...
• Google Code University: http://code.google.com/edu/paral...
• CS7960: http://www.cs.utah.edu/~jeffp/te...
• 6.829 http://ocw.mit.edu/courses/elect...
  
• Textbook: http://www.tacc.utexas.edu/~eijk...

Notes

• BitTorrent: http://en.wikipedia.org/wiki/Bit...

Other:

• Start with Lin, Data-Intensive Text Processing with MapReduce, ISBN 1608453421, http://www.umiacs.umd.edu/~jimmy...
• See Tom White, Hadoop: The Definitive Guide, ISBN 1449389732, http://www.amazon.com/Hadoop-Def...,  Sean Owen et al., Mahout in Action, ISBN 978193518268, http://www.manning.com/owen/
• HDFS under the hood:  http://assets.en.oreilly.com/1/e...
• Zookeper: http://highscalability.com/zooke...
• Download Hadoop (http://hadoop.apache.org/) and run some MapReduce jobs on your laptop in pseudo-distributed mode (see

MapReduce: What's the best way to come up to speed on MapReduce, Hadoop, and Hive? )

• Learn about Google technology stack (MapReduce, BigTable, Dremel, Pregel, GFS, Chubby, Protobuf, Snappy, Ganeti, Tenzing, Sawzall, BigQuery, F1, Spanner, Jingle, GCM, Google Talk, etc). (See

Google Research: What are the most interesting Google Research papers?
also  http://www.columbia.edu/~ak2834/... , http://www.cs.rutgers.edu/~muthu... , http://the-paper-trail.org/blog/...)

• Setup account with Amazon AWS/EC2/S3/EBS and experiment with running Hadoop on a cluster with large data sets (you can use Cloudera or YDN images, but in my opinion you can better understand the system if you set it up from scratch, using the original distribution). Watch the costs: http://www.networkworld.com/news...

• Try out Hadoop alternatives, specifically the minimalist frameworks such as BashReduce: http://github.com/erikfrey/bashr... and  CloudMapReduce: http://code.google.com/p/cloudma...  (see

What are some promising open-source alternatives to Hadoop MapReduce for map/reduce?)

• See Machine Learning: What are some good class projects for machine learning using MapReduce?
• Run Bryan Cooper's Cloud Serving Benchmark on AWS, compare Hbase vs
  Cassandra performance on a small cluster (6-8 nodes):  http://wiki.github.com/brianfran... also see Pete Warden's tests: http://petewarden.typepad.com/se... , Hbase book: http://hbase.apache.org/book.html
• Run LINPACK benchmark: http://www.datawrangling.com/on-...
• Run some experiments with MPI
  (http://www.mcs.anl.gov/research/...) try to implement a
  simple clustering algorithm
  (e.g http://en.wikipedia.org/wiki/K-m...) with MPI vs
  Hadoop/MapReduce and compare the performance, fault tolerance, ease of
  use etc.  Learn the differences between the two approaches, and when it
  makes sense to use each one.
• Check out Dongarra' papers: http://www.netlib.org/utk/people... , works by Gibbons: http://www.bell-labs.com/org/112... , Lamport: http://research.microsoft.com/en... , Blelloch: http://www.cs.cmu.edu/~guyb/pubs... , also see What are the seminal papers in distributed systems? Why?
• There is a new library called MPI-Mapreduce
  (http://www.sandia.gov/~sjplimp/m...) see how it works and how
   it compares to other MapReduce implementations
• Run some tests with Scalapack (http://www.netlib.org/scalapack/), try to port one of the routines to Hadoop, compare the performance and scalability. See how stability of numerical algorithms is evaluated: http://portal.acm.org/citation.c...
• Write your own simplified MapReduce runtime in C or any other programming language.
• Try xargs -P and GNU Parallel, also see What are some lesser known but useful Unix commands?
• Check out http://www.cascading.org/, http://clojure.org/ , http://www.bloom-lang.net/features/
• Learn about distributed hash tables (http://en.wikipedia.org/wiki/Dis... , http://www.linuxjournal.com/arti...), run some experiments with Paxos (Algorithm) http://the-paper-trail.org/blog/...,  Kademlia: http://en.wikipedia.org/wiki/Kad... , See Wolf Garbe's answers on Peer-to-Peer Technology. Also see Petar Maymounkov's answer to Computer Science Research: Have there been any new advances in distributed hash tables?
• Download Nutch (http://nutch.apache.org/) or Solr
  (http://lucene.apache.org/solr/), run a crawl on Wikipedia. Analyze the
  collected data with R (see item 2 above) or Python
  (http://www.nltk.org/)
• Write you own simplified crawler/indexer, test the performance and
  scalability, look at the Lucene source for ideas, look
  at http://infolab.stanford.edu/~bac... for inspiration. You
   can probably build it as a term project in either Information Retrieval
   or Search Engines course.
• Learn about prefix-sum: http://en.wikipedia.org/wiki/Pre... ,parallel
  matrix multiplication: http://www.cs.berkeley.edu/~yeli... ,streaming: http://infolab.stanford.edu/stream/ and
  BSP: http://en.wikipedia.org/wiki/Bul... , DSM: http://www.amazon.com/Distribute... and "The Use  of Name  Spaces in Plan 9"
• Check out Persistent Linda: http://www.google.com/#sclient=p... , if you find it interesting see Linda and Friends (an article by Sudhir Ahuja from 1986), also search for "Linda in Context",  "tuple space", Javaspaces, Gigaspaces. Read "How to write parallel programs: a guide to the perplexed" by Nicholas Carriero and David Gelernter: http://portal.acm.org/citation.c...
• Learn about Compute Unified Device Architecture (CUDA): http://www.amazon.com/CUDA-Examp... , Graphics Processing Unit and Field Programmable Gate Arrays accelerators, PlayStation 3 programming: http://en.wikipedia.org/wiki/Cel... , http://www.hotchips.org/hc21/mai...
• Pick one of the PGAS languages
  (http://en.wikipedia.org/wiki/Par...), e.g.
  X10 (http://en.wikipedia.org/wiki/X10..., go
  through the tutorials
  (http://ppppcourse.ning.com/forum...),
  run some HPC benchmarks (LU, FFT) and the examples (the streaming
  example in particular): see how it scales on a cluster/AWS, compare to
  sequential and Hadoop/MapReduce implementation, see what kind of
  performance/scalability gains it gives you on multicore boxes.
• Some good references on parallel programming: Herlihy& Shavit, The art
  of multiprocessor programming:
  http://www.amazon.com/Art-Multip... , Blelloch, Vector models for data-parallel computing:
  http://citeseerx.ist.psu.edu/vie... , Valiant, A bridging model for parallel computation:
  http://portal.acm.org/citation.c... ,Hillis & Steele, Data
  Parallel Algorithms: http://portal.acm.org/citation.c... , Miller & Boxer, Algorithms sequential and parallel: http://www.amazon.com/Algorithms... , Leighton, Introduction to Parallel Algorithms and Architectures: http://www.amazon.com/Introducti... , JaJa: Introduction to Parallel Algorithms: http://www.amazon.com/Introducti...
• You should probably start with Dijkstra,  Cooperating Sequential Processes: http://www.cs.utexas.edu/users/E... and Ben-Ari, Principles of Concurrent and Distributed Programming: http://www.amazon.com/Principles... (I have the older edition from 1982, which is an excellent intro)

• Take a course in Parallel Computer Architecture: http://www.eecs.berkeley.edu/~cu... , http://www.amazon.com/Parallel-C... , http://people.engr.ncsu.edu/efg/...
• Check out Cilk: http://software.intel.com/en-us/... and
  Matlab Parallel computing toolbox:
  http://www.mathworks.com/product...
• For some theoretical background on distributed algorithms, information decomposition and complexity see: Feldman et al., On the Complexity of Processing Massive, Unordered, Distributed Data: http://arxiv.org/abs/cs/0611108,  Traub, An introduction to information based complexity: http://octopus.library.cmu.edu/C... and Is Nancy Lynch's book still the best intro to distributed algorithms?
• Parallel Distributed Processing (PDP) by Rumelhart and PDP research group: http://www.amazon.com/Parallel-D...  - look up the computing architectures for Artificial Neural Networks, e.g. http://www.amazon.com/Parallel-A...
• Run some experiments with Weka (http://www.cs.waikato.ac.nz/ml/w...) or RapidMiner (http://rapid-i.com/), pick a simple algorithm and port it to MapReduce, see how it scales on a cluster/AWS
• Experiment with distributed 'NoSQL' data stores (Voldemort, Hbase, Redis, Tokyo, Cassandra etc).  Figure out what is CAP theorem all about
  (http://www.allthingsdistributed.... , http://www.cloudera.com/blog/201... ).
   Create a simple app with key-value or column-based store as a back-end.
   Import several GBs of interesting data into it and run some simple
  clustering/KNN algos (http://en.wikipedia.org/wiki/Clu..., http://en.wikipedia.org/wiki/Nea...).
   Optimize your algo to better utilize random access patterns, experiment
   with various tuning options. Build a frond-end visualization for the
  results (Check out Protovis or similar visualization
  package: http://vis.stanford.edu/protovis/)
• A good resource on 'NoSQL': Daniel Abadi's publications: http://cs-www.cs.yale.edu/homes/... and Varley, No Relation: The Mixed Blessings of Non-Relational Databases: http://ianvarley.com/UT/MR/Varle...
• Doozer: http://xph.us/2011/04/13/introdu...
• Learn about main-memory
  databases: http:YouTube: What is YouTube's architecture?//en.wikipedia.org/wiki/In-memory_database , http://scholar.google.com/schola..., http://monetdb.cwi.nl/ , http://hstore.cs.brown.edu/ , Microsoft Trinity - a graph database over distributed memory cloud: http://research.microsoft.com/en...
• Write a distributed hash table in C, here is a good reference: http://pdos.csail.mit.edu/papers... or use node: https://github.com/stbuehler/nod...
• Networking: http://www.amazon.com/Unix-Netwo... , http://www.amazon.com/TCP-Illust... , Network Programming: What are some good resources for learning about network programming?
• Write a distributed file system in C. See git for inspiration:  http://apenwarr.ca/log/?m=200801#31 , Frangipani http://portal.acm.org/citation.c... . For a good intro see the Tanenbaum's series: http://www.amazon.com/Distribute... , http://www.amazon.com/Modern-Ope... and http://www.stanford.edu/class/cs... , The Amoeba Distributed OS: http://www.cs.vu.nl/pub/amoeba/a...

• Graph databases, etc: http://nosql-database.org/ , http://www.graph-database.org/ , GraphLab: http://graphlab.org/
• Facebook Engineering: What is Facebook's architecture? , What are the most interesting Facebook Data papers/projects?
• Hadoop/Hbase at Facebook: http://borthakur.com/ftp/Realtim...
• YouTube: What is YouTube's architecture?
• Justin.tv: How does justin.tv work?
• Scalability: How does Heroku work?
• Netflix: What is Netflix's architecture?
• Hulu: What is Hulu's architecture?
  
• eBay: What is eBay's architecture?
• Dropbox: What is Dropbox's architecture?
• What are the core technologies that Twitter uses for their platform and what is the Twitter Macro architecture?
• LinkedIn SNA: http://sna-projects.com/sna/
• What is LinkedIn's database architecture like?
• Quora Infrastructure: How does LiveNode work?
• Scaling LiveJournal: http://danga.com/words/2007_06_u...
• Content Delivery Networks: How does Akamai CDN work?
• GitHub architecture:  https://github.com/blog/530-how-...
• Twitter Rainbird: http://www.slideshare.net/kevinw...,  http://www.slideshare.net/nkalle...
• Yahoo! S4: https://github.com/s4/core , http://docs.s4.io/
• IBM Infosphere streams/System S: http://www-01.ibm.com/software/d...
• BackType Storm: http://news.ycombinator.com/item...

• Octobot, a distributed task queue worker: http://octobot.taco.cat/
• F* by Microsoft: http://research.microsoft.com/en...

• Google Realtime Search: Is real-time search fundamentally different from the various search services Google already provides?

• HN thread on the architecture of backend systems: http://news.ycombinator.com/item...
• The secrets of Node's success: http://radar.oreilly.com/2011/06...
• Druid: A Distributed, In-Memory OLAP Store: http://metamarketsgroup.com/blog... (some dissing here: http://news.ycombinator.com/item...)
• FathomDB response to AWS outage: http://news.ycombinator.com/item...
• Google Ganeti - Cluster-based virtualization management software: http://code.google.com/p/ganeti/ , http://k1024.org/~iusty/papers/i...
• Google GO: http://www.theregister.co.uk/201...
• Erlang/OTP: http://learnyousomeerlang.com/co...
• Cloud Haskell: http://research.microsoft.com/en...
• NASA Nebula: http://nebula.nasa.gov/
• Platform MR: http://www.platform.com/Products...
• Fast 2011: http://www.usenix.org/events/fas...
• The history of consensus: http://betathoughts.blogspot.com... (via http://the-paper-trail.org )
• Distributed Linked List: http://www.google.com/search?scl...
• GIbbons, Synopsys Data Structures For Massive Data Sets: www.cs.princeton.edu/courses/archive/spring04/cos598B/bib/GibbonsM-syn.pdf
• Lock-Free Linked Lists and Skip Lists: http://www.cse.yorku.ca/~ruppert...
• Maekawa's lock: http://www.google.com/#sclient=p...
• Crossbow, searching for SNPs with cloud computing: http://www.biomedcentral.com/con...
• Distributed Caching: Hazelcast, Ehcache, Terracotta (company), Memcached, Oracle Coherence
• Bitcoin, A Peer-to-Peer Electronic Cash System: www.bitcoin.org/bitcoin.pdf
• Distributed computing with JS: BitCoin miner: http://news.ycombinator.com/item..., MapRejuice: https://github.com/ryanmcgrath/m... , http://www.igvita.com/2009/03/03...
• OpenCirrus - Cloud Computing Research Testbed: https://opencirrus.org/content/r...
• Antonio Piccolboni, A Comparison of Eight MapReduce
  Languages: http://www.dataspora.com/2011/04...
• Caching and processing 2TB in memory with Hazelcast: http://highscalability.com/blog/...
• Dapper: a Large-Scale Distributed Systems Tracing Infrastructure: http://static.googleusercontent....
• On the performance of distributed lock-based synchronization: http://portal.acm.org/citation.c...
• Tonika: social routing with organic security: http://pdos.csail.mit.edu/~petar...
• http://www.linuxvirtualserver.org/
• It's time for low latency: http://www.matt-welsh.blogspot.c...
• OS Research Wanted: http://surriel.com/research_wanted
• FlightPath: Obedience vs. Choice in Cooperative Services:  http://www.usenix.org/event/osdi...
• Piccolo: Building Fast, Distributed Programs with Partitioned Tables: http://piccolo.news.cs.nyu.edu/p...
• CIEL: a universal execution engine for distributed data-flow computing: www.usenix.org/event/nsdi11/tech/full_papers/Murray.pdf
• Directed Edge: On building a stupidly fast graph database: http://blog.directededge.com/200...
• What is the best tutorial for Python's Twisted framework?
• Node.js: What are the best resources to learn Node.js?
• Parallelism /= Concurrency:  http://ghcmutterings.wordpress.c...
• PyCon 2011: Handling ridiculous amounts of data with probabilistic data structures: http://blip.tv/pycon-us-videos-2...
• Go (programming language) at Heroku: http://blog.golang.org/2011/04/g...
• Meijer & Lamport,  Mathematical Reasoning and Distributed Systems: http://channel9.msdn.com/Shows/G...
• Concurrency's Shysters: http://blogs.oracle.com/bmc/entr...
• Horton: Online Query Execution On Large Distributed Graphs: http://www.graph-database.org/20...
• DataDomain: http://www.datadomain.com/

• WebRTC: https://sites.google.com/site/we...
• Bloom: http://www.bloom-lang.net/ and http://boom.cs.berkeley.edu/pape...
• Jini tutorial: http://jan.newmarch.name/java/ji...
• java distributed cache for low latency, high availability: http://stackoverflow.com/questio...
• Scalable, Distributed Data Structures for Internet Service Construction (2000):  http://usenix.org/events/osdi200...
• Cloud Programming: From Doom and Gloom to BOOM and Bloom: http://neilconway.org/talks/boom...

• SEDA: An Architecture for Highly Concurrent Server Applications: http://www.eecs.harvard.edu/~mdw... (http://matt-welsh.blogspot.com/2... )
• Scalable Network programming: http://bulk.fefe.de/scalable-net...
• Protocol Buffers: http://news.ycombinator.com/item...

• Distributed Systems: What is the future of file systems?

• What are some current directions in operating system research?
• Distributed Systems: What are the best resources for learning about distributed file systems?
• How do I approach building a distributed queue architecture?
• What are some good resources for learning about data compression? Why?
• Information Retrieval: What are some good resources to get started with Information Retrieval? Why?
• What are good resources to learn about search engine architecture?
• What are the good resources to learn about distributed, scalable, robust software architecture/infrastructure building?
• What are some common approaches to error aggregation, alerting, and analysis in distributed systems?
• What are some good research papers and articles on fault-tolerant systems design?
• Big Data: What are the most influential papers in the world of big data? Why?
• Large Scale Learning: What are some introductory resources for learning about large scale machine learning? Why?
• Computer Science Research: Which CS areas have the most low-hanging fruit for research?
• Big Data: Why the current obsession with "big" data?
• Distributed Systems: Which conferences are the best to follow for Distributed Systems?
• What are the best recommended research topics on databases according to edge technologies and recent research trends?
• Distributed Systems: What are the most interesting research projects related to the management of distributed systems?
• TCP/IP: What are some high performance TCP hacks?
• Pike, Systems Software Research is Irrelevant: http://herpolhode.com/rob/utah20...
• Concurrency in Go (programming language): http://golang.org/doc/effective_...
• Communicating Sequential Processes (CSP): http://www.usingcsp.com/
• Scalable Joins: http://research.microsoft.com/en...
• Kestrel, tiny queue system based on starling, in scala: https://github.com/robey/kestrel
• Disruptor - concurrent programming framework: http://code.google.com/p/disruptor/
• DataTurbine streaming engine: http://www.dataturbine.org/
• The Task Parallel Library (TPL) in .NET: http://msdn.microsoft.com/en-us/...
• A crash course on modern hardware: http://www.infoq.com/presentatio...
• Infinispan data grid on top of JGroups: http://www.jboss.org/infinispan
• Memcached distributed cache on top of Jgroups: http://www.jgroups.org/memcached...
• Scalable Application Layer Multicast: http://pages.cs.wisc.edu/~suman/...
• Hadapt: Efficient Processing of Data Warehousing Queries in a split execution environment: http://portal.acm.org/citation.c...
• Danga's open source projects: http://danga.com/
• Hadoop on MPI: http://hadoopbi.com/index.php/te...
  
• Hadoop on Pallet: http://sritchie.github.com/2011/...
• Oracle Grid Engine: http://en.wikipedia.org/wiki/Ora...
• Ejabberd:  a scalable XMPP instant messaging server: http://www.ejabberd.im/
• Cheetah - Circuit-switched High-speed End-to-End Transport ArcHitecture:
  http://www.ece.virginia.edu/chee...
• PVM: http://www.snakebytestudios.com/...
• Systems at ETH Zurich: http://www.systems.ethz.ch/resea...
• OpenCL: http://www.khronos.org/opencl/
• Cloud Haskell: http://research.microsoft.com/en...
• Concurrent programming in Erlang: http://www.erlang.org/erlang_boo...
• Concurrent programming in Occam 2: http://www.amazon.com/Programmin...
• Concurrent and Real-Time Programming in Ada: http://www.amazon.com/Concurrent...
• Distributed Programming in Ruby: http://www.amazon.com/dp/0321638...
• REST: http://rest.elkstein.org/
• Gearman: http://gearman.org
• Drizzle: https://launchpad.net/drizzle
• Distributed logging with syslog:  https://wiki.archlinux.org/index...
• Logs are streams, not files: http://adam.heroku.com/past/2011...
• Utilizing Redis in distributed Erlang systems (Heroku):  http://erlang-factory.herokuapp....
• MS Command Shell: http://arstechnica.com/business/...
• MS PowerShell: http://en.wikipedia.org/wiki/Win...
• DRb - Distributed Ruby: http://segment7.net/projects/rub...
• God - The Ruby Framework for Process Management: https://github.com/mojombo/god
• Taco Bell programming: http://teddziuba.com/2010/10/tac...
  
• Rush - the Ruby Shell: http://rush.heroku.com/
• CloudCrowd: https://github.com/documentcloud...
• Coda: http://www.coda.cs.cmu.edu
• Tenzing: http://research.google.com/pubs/...
• GNTP: http://www.growlforwindows.com/g...
  
• CycleCloud : http://blog.cyclecomputing.com/2...
  
• GNU Parallel: http://www.gnu.org/software/para...
  
• Torque: http://www.adaptivecomputing.com...
• Chef: http://www.opscode.com/chef/
• Dempsy - Nokia's Distributed Elastic Message Processing System: http://dempsy.github.com/Dempsy/
• F1 -The Fault-Tolerant Distributed RDBMS Supporting Google's Ad Business: http://research.google.com/pubs/...
• Galaxy - a distributed in-memory data grid by Parallel Universe: http://blog.paralleluniverse.co/...
  
• Spanner: Google's Globally-Distributed Database:  http://research.google.com/archi...
• Jingle: http://code.google.com/p/libjingle/
• Paxos Made Live: http://www.eecs.harvard.edu/cs26...
• TeleHash: https://github.com/quartzjer/Tel...
• Adobe RTMP: http://en.wikipedia.org/wiki/Rea...
• UPnP: http://en.wikipedia.org/wiki/Uni...
  
• IP multicast: http://en.wikipedia.org/wiki/IP_...
• Reliable multicast: http://en.wikipedia.org/wiki/Rel...
• JGroups: http://www.jgroups.org/
• LDPC Codes: http://en.wikipedia.org/wiki/Low...
• Erasure codes: see the last chapter in D.J.C MacKay's  http://www.inference.phy.cam.ac.... (note that some of these are heavily patented, e.g. http://www.inference.phy.cam.ac.... )
• Blaze: Next-gen NumPy on which to build out-of-core and distributed algorithms: https://speakerdeck.com/sdiehl/b...
• Spark cluster computing by  UC Berkeley AMPLab
  
• NSQ by bitly | ♥ your bitmarks: NSQ: realtime distributed message processing at scale
• Celery: Distributed Task Queue
  
• Go'Circuit by Tumblr: Paradigm for developing and sustaining Big Data apps
• List of popular backend stacks: ragingwind/backend-architectures.md
  
• CRAN Task View: High-Performance and Parallel Computing with R
  
• Apache Accumulo by NSA
  
• alternative-internet
• OpenFlow for programmable networks: Page on Openflow, switch spec: http://archive.openflow.org/docu...
• On data center scale, OpenFlow, and SDN
• Docker, The linux container engine , and one use case at Etsy: LXC - Running 14,000 tests per day and beyond! (Part 1)
• Apache Mesos (A Platform for Fine-Grained Resource Sharing in the Data Center, tech report: Page on Berkeley )
  
• Apache Spark: Lightning-Fast Cluster Computing via https://news.ycombinator.com/ite...
  
• Google's MillWheel: Fault-Tolerant Stream Processing at Internet Scale: Page on Googleusercontent
• Google's Sibyl, a distributed learning system: www.magicbroom.info/Papers/Ladis10.pdf‎
• Berkeley amplab - SQL Benchmark: Redshift vs Hive vs Impala vs Shark : Big Data Benchmark based on Page on Brown
• The Log: What every software engineer should know about real-time data's unifying abstraction | LinkedIn Engineering by Jay Kreps (also Kafka: a Distributed Messaging System for Log Processing )

(This is a live list. Edits and additions welcome)

View Answer on Quora

Why is ETL associated mostly with data warehousing? Isn't ETL a procedure that eventually may apply when loading raw data to a transactio...

Why is ETL associated mostly with Data Warehousing?
Isn't ETL a procedure that eventually even may apply when loading raw data to a transactional databse?

View Question on Quora

Eventual consistency in simple terms

Unless you live under a rock, you know 15 minutes can save you 15% and big data is going to make life super awesome!

I don't know if GEICO has consistency issues but big data does and I get the feeling the IT media and software vendors are trying to hide that under a rock.

Big data does a few things really well.  Specifically, write data and scale.  Sounds awesome, huh?  Well, hold on.  In order to do this it has many, many copies of the data stored all over the place.  Sometimes the copy A is updated and sometimes copy B is updated.  The point is to be able to write data at amazing scale and speed and distributing the data all over the place achieves this very difficult task.

But, but ...

Don't worry, eventually these copies of data are synchronized to represent a single version of the truth.

But, but ...

The word eventually scare you?  It should.  You see, in order to write data at all times and scale infinitely, something had to give.  That something is the synchronization of distributed data.

Put simply, big data distributes multiple copies of a database, writes to whichever copy is available and synchronizes the data when it can.

What the ...

I know, right?  I'm curious if there is a distributed system out there that has ever reached consistency.  If the point is to write at all costs because you have such a high volume of updates, when is there time to synchronize?

For those who prefer a more technical definition of eventual consistency:

Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value.[1]

How does this necessity of invention sit with you?

[polldaddy poll=7768766]

Microsoft Data Quality Services: Market Impact?

I have been getting my feet wet with Microsoft's Data Quality Services and as I read more and more I cannot help but think about how this feature introduction will impact the DQ market.  I know there will be some that scuff at this suggestion, but I think there a ripple forming in Seattle (and I am not talking about Richard Sherman) that is going to stir the market.

Here is why I think Microsoft has the potential to disrupt the DQ market ...

•  most DQ solutions on the market cost more than the SMB market can afford


• they require skill sets that are hard to come by and demand considerable compensation


• many SMBs own MS SQL Server for some, if not all, their database needs

what do you think?

[polldaddy poll=7739976]

 

 

relationship fail

trust me

this is not a relationship strategy

Would you read my Big Data book?

As you my know, I am pretty critical of the "big data" hype in the information technology media.  One of my criticisms is the very term "big data".  Frankly, I hate the term as much as I hate the outrageous claims that are out there.

I prefer to talk about distributed systems which is really the differentiator in this space, in my humble [ref] degrees of humility can easily be argued[/ref] opinion.

As I researched distributed systems (DS), I came across a topic that struck a chord with my passion for data quality.  The topic was the CAP Theorem and notion of eventual consistency.  At this point, I am becoming obsessed with the notion of eventual consistency.

As a result, I am convinced I need to write a book on the topic and its implications on deploying distributed systems to a business and its decisions.

However, before I get too far down the line, I want to know is ...

would you read my book about eventual consistency and how that would impact a distributed system's ability to aid in the process of making business decisions?

Please vote and let me know!

[polldaddy poll=7427832]

MySQL: loading data from an external file

Path of least resistance

When you need to get something done quickly, the community edition of MySQL is a great option.   It has just what you need in terms of a management studio and requires very little setup.

Yesterday I was in that exact circumstance.  I have some data files that I need to analyze and not much time to do it.  So I downloaded the community edition of MySQL (found here).

Data time!

As I said, I needed to get some analysis done quickly, so I created a few tables with your basic ANSI  create table statements and I was ready to load my data.

I was loading data from delimited sources not located in the default MySQL data directory so I had to include a few wrinkles in my load data statement. By the way, load data is the command for loading data from a file into a MySQL table.

The wrinkles

Since I was loading data from an external file I had to include the following statement in my load data statement:
 Load data LOCAL INFILE 'INSERT FILE PATH HERE'
I was using a comma delimited file so I had to include the following statement as well:
FIELDS TERMINATED BY ','
And finally, I had column headers in external file so the following statement was necessary:
IGNORE 1 LINES
The load script, when complete, looked like this:
Use SCHEMA NAME;
Load data LOCAL INFILE Load data LOCAL INFILE 'INSERT FILE PATH HERE'
into table TARGET TABLE HERE
FIELDS TERMINATED BY ','
IGNORE 1 LINES;

Nothing too magical, but I wanted to pass it along and, hopefully, save some of you the time in looking up the statements.

More information on MySQL load data parameters can be found here