[SPARK-24907][SQL][WIP] Migrate JDBC DataSource to JDBCDataSourceV2 Read using DataSourceV2 API #21861
Conversation
|
Thanks for your PR, but we plan to propose new changes in DataSourceV2 APIs very shortly. |
|
@gatorsmile Got you. I will update the implementation after DataSourceV2 API changes. |
tengpeng
changed the title
|
Any update on this? @tengpeng |
|
@xianyinxin I have not followed the progress on the V2 APIs for a while, but I am still interested in implementing the V2 APIs for JDBC. I am OK to co-author if you are interested in this PR. |
|
@tengpeng i can also help co-author here. Should we start fixing this branch or create a new branch based on latest V2? I have a local branch with with basic scaffolding for read and write paths based on latest V2. I have not push that out as yet, i can if you want to start using this. FYI to @xianyinxin @gaborgsomogyi who have also expressed their interest in authoring this. |
|
Glad you have already some work on this @shivsood . Since this is a private repo which is not allowed multi people push each one's code, my suggestion is folk this repo, cherry-pick @tengpeng 's commit, rebase it based on current V2 code, apply your fix, and push your code to your own branch, and then submit pr @shivsood . Would you agree? @shivsood @tengpeng |
|
Sound's ok to me. I will try try to get this out soon.
Regards,
Shiv
…On Mon, Jul 15, 2019 at 9:10 PM Xianyin Xin ***@***.***> wrote:
Glad you have already some work on this @shivsood
<https://github.com/shivsood> . Since this is a private repo which is not
allowed multi people push each one's code, my suggestion is folk this repo,
cherry-pick @tengpeng <https://github.com/tengpeng> 's commit, rebase it
based on current V2 code, apply your fix, and push your code to your own
branch, and then submit pr @shivsood <https://github.com/shivsood> .
Would you agree? @shivsood <https://github.com/shivsood> @tengpeng
<https://github.com/tengpeng>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#21861?email_source=notifications&email_token=AAMBQMPPRI7V4YNRD4SRTDLP7VCZ5A5CNFSM4FLSZXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ7UFZA#issuecomment-511656676>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAMBQMPC5HGXJHKJUKSBHU3P7VCZ5ANCNFSM4FLSZXOQ>
.
|
|
@xianyin Xin <xianyin.xxy@alibaba-inc.com> if i just create Work In
Progress PR . Is it possible for all of us to collaborate on that PR? I am
wondering if forking my repo is must have?
@tengpeng <https://github.com/tengpeng> - are you ok with this approach?
Regards,
Shiv
On Mon, Jul 15, 2019 at 9:21 PM Shiv Prashant Sood <shivprashant@gmail.com>
wrote:
… Sound's ok to me. I will try try to get this out soon.
Regards,
Shiv
On Mon, Jul 15, 2019 at 9:10 PM Xianyin Xin ***@***.***>
wrote:
> Glad you have already some work on this @shivsood
> <https://github.com/shivsood> . Since this is a private repo which is
> not allowed multi people push each one's code, my suggestion is folk this
> repo, cherry-pick @tengpeng <https://github.com/tengpeng> 's commit,
> rebase it based on current V2 code, apply your fix, and push your code to
> your own branch, and then submit pr @shivsood
> <https://github.com/shivsood> . Would you agree? @shivsood
> <https://github.com/shivsood> @tengpeng <https://github.com/tengpeng>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#21861?email_source=notifications&email_token=AAMBQMPPRI7V4YNRD4SRTDLP7VCZ5A5CNFSM4FLSZXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ7UFZA#issuecomment-511656676>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAMBQMPC5HGXJHKJUKSBHU3P7VCZ5ANCNFSM4FLSZXOQ>
> .
>
|
|
@shivsood OK for me. |
|
@xianyin Xin <xianyin.xxy@alibaba-inc.com> @tengpeng
<https://github.com/tengpeng> - I added you as collaborators. Pushed by
changes dsv2_jdbc branch in this repo. I have *not* created a <draft> PR as
this is just scaffolding code. Also experimenting with this collaboration
model :-)
@tengpeng <https://github.com/tengpeng> if you are ok , can u cherry-pick
your changes to dsv2_jdbc branch? or i can do that as i get to work
tomorrow.
Regards,
Shiv
On Mon, Jul 15, 2019 at 9:37 PM Shiv Prashant Sood <shivprashant@gmail.com>
wrote:
… @xianyin Xin ***@***.***> if i just create Work In
Progress PR . Is it possible for all of us to collaborate on that PR? I am
wondering if forking my repo is must have?
@tengpeng <https://github.com/tengpeng> - are you ok with this approach?
Regards,
Shiv
On Mon, Jul 15, 2019 at 9:21 PM Shiv Prashant Sood ***@***.***>
wrote:
> Sound's ok to me. I will try try to get this out soon.
>
> Regards,
> Shiv
>
> On Mon, Jul 15, 2019 at 9:10 PM Xianyin Xin ***@***.***>
> wrote:
>
>> Glad you have already some work on this @shivsood
>> <https://github.com/shivsood> . Since this is a private repo which is
>> not allowed multi people push each one's code, my suggestion is folk this
>> repo, cherry-pick @tengpeng <https://github.com/tengpeng> 's commit,
>> rebase it based on current V2 code, apply your fix, and push your code to
>> your own branch, and then submit pr @shivsood
>> <https://github.com/shivsood> . Would you agree? @shivsood
>> <https://github.com/shivsood> @tengpeng <https://github.com/tengpeng>
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <#21861?email_source=notifications&email_token=AAMBQMPPRI7V4YNRD4SRTDLP7VCZ5A5CNFSM4FLSZXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ7UFZA#issuecomment-511656676>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AAMBQMPC5HGXJHKJUKSBHU3P7VCZ5ANCNFSM4FLSZXOQ>
>> .
>>
>
|
|
@shivsood Accepted. Seems good. |
|
@tengpeng Tried cherry-picking your change. It's not a direct cherry pick. Directory paths have changed and updated files also have changes. Some of your work in JDBCOptionsV2 , JDBCDataSourceV2.scala and Test is very much relevant, but needs to be adapted into the new baseline. Suggest you take a shot at manually merging and make a commit? |
|
@gaborgsomogyi have added you as collaborator |
|
@shivsood thanks for considering me as collaborator. Since I'm mainly in the streaming area my main intention is to implement Structured Streaming source/sink based on the SQL implementation. Additionally you've guys already picked this up so I'm contributing in this effort with review comments. |
@gaborgsomogyi Thanks. Reviews would be a great help indeed. Also look forward to your streaming source/sinks work. |
|
@tengpeng @xianyinxin @gaborgsomogyi Feel free to comment/contribute as relevant. Thanks. |
|
Can one of the admins verify this patch? |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
What changes were proposed in this pull request?
(After the update of DataSourceV2 API, this PR will be updated)
Changes: Migrate JDBC DataSource to JDBCDataSourceV2 Read using DataSourceV2 API
How was this patch tested?
New unit tests.