Dynamic partitioning in google cloud dataflow? -

-

i'm using dataflow process files stored in gcs , write bigquery tables. below requirements:

  1. input files contain events records, each record pertains 1 eventtype;
  2. need partition records eventtype;
  3. for each eventtype output/write records corresponding bigquery table, 1 table per eventtype.
  4. events in each batch input files vary;

i'm thinking of applying transforms such "groupbykey" , "partition", seems have know number of (and type of) events @ development time needed determine partitions.

do guys have idea partitioning dramatically? meaning partitions can determined @ run time?

why not loading single "raw" bigquery table , using bigquery api determine different number of events , export each event type own table (e.g., via https://cloud.google.com/bigquery/bq-command-line-tool#createtablequery) or api call?

if input format simple, can without using dataflow @ , more cost efficient.


Comments

Post a Comment

Popular posts from this blog

node.js - How to mock a third-party api calls in the backend -

-
i'm testing application mocha , supertest, test like var request = require('supertest'), app = require('./bootstrap.js'); describe('...', function() { it('...', function() { request(app) .get('/some/url') // ... }); }); the bootstrap.js real application before listen ports all works perfect have add calls third-party api , testing slow so when do, test example takes 5 seconds request(app) .get('/my/endpoint/that/makes/call/others') // ... how can make fake calls during test real when running application? you can use nock purpose.
Read more

node.js - Why do I get "SOCKS connection failed. Connection not allowed by ruleset" for some .onion sites? -

-
i'm experimenting node , socks5-https-client . reason, tor hidden service ( .onion ) sites return connection error. for example, connecting duckduckgo ( 3g2upl4pq6kufc4m.onion ) works , returns html. however, connecting pirate bay ( uj3wazyk5u4hnvtk.onion ) or torch ( xmh57jrzrnw6insl.onion ) returns... error: socks connection failed. connection not allowed ruleset. what error mean? how can avoid it? here's code reproduce it: var shttps = require('socks5-https-client'); shttps.get({ hostname: '3g2upl4pq6kufc4m.onion', path: '', sockshost: '127.0.0.1', socksport: 9150, rejectunauthorized: false }, function(res) { res.setencoding('utf8'); res.on('readable', function() { console.log(res.read()); // log response console. }); }); the error seems caused 0x02 value in field 2 of server response. in summary the servers you're failing access don't support ...
Read more

matlab - 0-by-1 sym - What do I need to change in order to get proper symbolic results? -

-
i solve system of equations symbolically beta1 , beta2 , , beta3 . defined variables follows , set equation system: w1 = sym('w1', 'real'); w2 = sym('w2', 'real'); me1 = sym('me1', 'real'); me2 = sym('me2', 'real'); btm1 = sym('btm1', 'real'); btm2 = sym('btm2', 'real'); mom1 = sym('mom1', 'real'); mom2 = sym('mom2', 'real'); gamma = sym('gamma', 'real'); t = sym('t', 'real'); beta1 = sym('beta1', 'real'); beta2 = sym('beta2', 'real'); beta3 = sym('beta3', 'real'); nt = sym('nt', 'real'); r1 = sym('r1', 'real'); r2 = sym('r2', 'real'); syms e1 e2 e3 real b = [1/t * (1 + ( w1 + 1/nt * beta1 * me1 + beta2 * btm1 + beta3 * mom1 ) *r1 ) ^(-gamma) * ( 1/nt * me1 * r1 ) + 1/t * (1 + ( w2 + 1/nt * beta1 * me2 + beta2 * btm2 + beta3 *...
Read more