Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overall Testing Story for Office Add-ins #584

Closed
LanceEa opened this issue Sep 20, 2017 · 50 comments · Fixed by #3079
Closed

Overall Testing Story for Office Add-ins #584

LanceEa opened this issue Sep 20, 2017 · 50 comments · Fixed by #3079
Assignees
Labels
Area: add-ins Feedback on add-ins content that's relevant to multiple Office hosts. Status: in backlog Issue is being tracked in the backlog but timeline for resolution is unknown Type: doc request Request for new documentation or updates/enhancements to existing documentation

Comments

@LanceEa
Copy link

LanceEa commented Sep 20, 2017

This potentially is a User Voice but thought I would start here

TL;DR

  • Challenge 1: No documentation, or direction on mocking Office-Js Api's for testing.
    I know there are many permutations and opinions on tooling and testing frameworks so documenting them all is not feasible. But, potentially providing a mock or at least a small write up showing some samples I think would help.

  • Challenge 2: No easy way to test against multiple platforms/versions of Excel
    It would be nice if there was an easier way to test against multiple platforms. Similar to BrowserStack, SauceLabs and Xamarin Test Cloud or at something like what the Edge Team did with Virtual Machines. This is probably more of a User Voice but also could benefit from a short write-up giving some guidance.

Issue:
There is no documentation on how to best test add-in code. I understand that is fairly broad so let me explain.

I'm developing my add-in with Angular which has a great Unit and E2E integration testing story. Since the add-ins is web based I can leverage the Angular teams opinions on testing and if someone was using some other framework they could follow the guidelines set by that framework. But, where it gets a little bit dicey is when you start interacting with Office-js API's.

So, what I did was first focus completely on the UI and aspects of add-in that do not require office-js. This way I can follow my normal developer workflow (Chrome Debug tools, HMR code refreshes, automated unit testing,etc...). Then I introduced the Office API's and at this point I have avoided writing any tests for those functions. To me that is the first challenge because there is no examples, no mocks and in general no direction on how to best test these. I recognize I could mock/spy on the office api's but the setup to do that would be fairly painful (I recognize someone has to feel the pain at least once even if a mock was provided 😄 ).

The other challenge is when we need to actually test the add-in within Excel. I have to physically test on my Mac, my Windows 10 PC and with different version of Excel. The number of combinations of OS/Excel version is definitely not controllable, especially with external users (outside our company). So, I would love to get some direction here.

@lindalu-MSFT
Copy link
Collaborator

Hi @LanceEa! I'm on the Add-ins team and want to apologize for the delayed response. We got buried preparing for and then attending Ignite. It's a little late tonight to ask a content developer on my team to repspond, but I saw this and wanted to thank you for taking the time to write and point out our content gap when it comes to testing add-ins. I'm going to pass this on to our manager @Reezaali to address. Cheers, Linda

@lalitkapoor
Copy link

Thanks @lindalu-MSFT. Can’t wait to hear more on this topic from @Reezaali!

@lindalu-MSFT
Copy link
Collaborator

Hi @LanceEa Have you had a chance to look at the topics in https://aka.ms/Kqj3uz? This documentation node has several topics on testing and debugging. Outlook has some topics as well here: https://docs.microsoft.com/en-us/outlook/add-ins/testing-and-tips.

I noticed someone asked this question in Stack Overflow: https://aka.ms/B1o3qn

linda

@LanceEa
Copy link
Author

LanceEa commented Nov 1, 2017

@lindalu-MSFT - I figured most people were busy with Ignite :) so no worries.

I look forward to hearing more on this because it definitely is still a challenge and I believe it has an opportunity to increase developer adoption of Office Add-ins if these challenges can be addressed.

As for the links you sent, most of them are just how to do manual testing by side-loading the add-ins. This works fine for manual testing an add-in and for early distribution but isn't quite what I was referring to above.

Challenge 1 above is more about automated testing and continuous integration scenarios where when I commit changes in my code repository, I then need to run automated unit test and integration test, etc... before my changes can be deployed. You can mock/create spies for office-js in unit tests but there is a lot to mock so this becomes tedious. It is also very difficult to run end-to-end/integration testing.

Challenge 2 goes along with the last point about difficult to do end-to-end/integration testing because there is no easy way to test on various versions of excel (i.e. Office 2013, Office 2016 Slow Channel, Office Online, Office for mac, etc...). The best we have is the requirement sets which is more similar in my mind to feature detection. Kind of like what Modernizr(https://modernizr.com/) has offered web developers for a few years. But, if a user comes to me and says I'm seeing issue "XYZ" and I'm on Office 2013. There is no easy way for me to test this because my Mac & PC both have the latest versions. Which means I need to have other PC's or virtual environments installed with various versions. Also, that user might not be on the latest patch release. Our support response could be update Office then come back to use but unfortunately reality is that won't fly :). So, I would love to see a better overall story for this.

Let me know if anyone wants to kick around ideas. More than willing to go into more details.

@kbrandl kbrandl self-assigned this Jan 5, 2018
@kbrandl
Copy link
Contributor

kbrandl commented Jan 12, 2018

@LanceEa -- thanks so much for the providing such a thoughtful and thorough explanation of the challenges you're encountering re the testing of Office Add-ins. I understand completely and wanted to let you know that the Office Add-ins documentation folks are aware of this documentation gap and are actively working to address it (although the timeframe for publication of these docs is still TBD). In the meantime, I'll see if I may be able to identify someone on the product team that would be able to provide you with some guidance re some of the challenges that you've encountered.

@joekrump
Copy link

joekrump commented Mar 15, 2018

@kbrandl Any update on this?

I'm also curious to know if there a recommended end-to-end testing tool that works for testing the web version of Outlook Add-ins?

@lindalu-MSFT
Copy link
Collaborator

@LanceEa @joekrump We do have a few new topics in our documentation:

Under the Office Add-ins test and debug section, here's a new topic:

Troubleshoot user errors with Office Add-ins

There's also Debug Office Add-ins in Visual Studio and Debug add-ins using F12 developer tools.

@kbrandl
Copy link
Contributor

kbrandl commented Mar 16, 2018

@LanceEa @joekrump apologies for the slow response on this; unfortunately I don't yet have any new information to share. I'll follow up again with the folks that I initially pinged about this issue, and you should see this thread updated with new information as it becomes available.

@sureshjoshi
Copy link

For what it's worth (and admittedly, might not be worth a lot), I've had decent success in using Bob Martin's Clean architecture (https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html) for developing my plugins - in order to make them somewhat testable.

This is entirely commentary, not really solving any of @LanceEa's problems - just might be useful for someone else down the line.

I won't re-hash the details, other than to note that most of my business logic (80+%) occurs in API/Mock-free code (use cases) which are easily tested with the standard JS unit test tools (ie. Mocha-ish) in a 'domain' layer.

Then I have a set of tests and mocks for my 'data' layer, which contains database-esque components - and these are also unit testable and injected into higher layers. Many 3rd party dependencies which might need to get flipped back out can go here.

The next component is the UI layer, which is UI-testable in any headless web-browser (interactions with the Word document are spied out).

Lastly comes the interaction with the Word document itself, which I haven't solved - and is kinda the crux of this issue :)

I've played around with automated testing that simulates clicks, then exports documents to be compared offline - but it's kinda meh, as it falls prey to the classic problems with click-centric testing.

@kbrandl
Copy link
Contributor

kbrandl commented Mar 30, 2018

@LanceEa thank you for your patience as I've tried to track down more information for you regarding the testing-related challenges that you've described in this thread.

  • Re challenge 1 (no guidance available about end-to-end testing of Office Add-ins), I've confirmed with the Office Add-ins documentation team that there's already a work item in their backlog to address this documentation gap. Timeline for publication of this guidance is currently still TBD, but it's in the queue.

  • Re challenge 2 (no easy way to test against multiple platforms/versions of Excel), you were correct in your initial instinct that this suggestion would be a good candidate for UserVoice. Can you please post it there (if you haven't already done so), so that other customers can add their votes for it in the days ahead?

Thanks again for your patience and for providing such a detailed write-up of the testing-related challenges that you've been facing with Office Add-ins. I'll leave this issue open to track the request for additional documentation about end-to-end testing.

@maxpaj
Copy link

maxpaj commented Apr 11, 2018

+1

Need to be able to test our Outlook Add-in on Outlook 2013 for customers that have slow moving IT departments. At the moment a customer is experiencing issues and I can't test anything on my end. What do I do?

@kbrandl kbrandl assigned Reezaali and unassigned kbrandl Apr 11, 2018
@carlosconnected
Copy link

carlosconnected commented May 16, 2018

@lindalu-MSFT @kbrandl is there a way that you folks could provide office.js as a module that could be mocked through webpack's NormalModuleReplacementPlugin. The purpose would be to run an Add-In as a standalone webapp, so standard integration tests could be run using frameworks like Cypress or Selenium?

I'm working on a Add-In implemented on angularjs and I have not been able to achieve what I just proposed.

@carlosconnected
Copy link

carlosconnected commented May 17, 2018

I think if the restriction of having the office.js coming from a CDN is changed and it's instead modularized, it will be possible to run integration tests by having Add-ins running as standalone apps just by replacing the module with an office mock module.

@LanceEa
Copy link
Author

LanceEa commented May 25, 2018

@kbrandl - Sorry for the slow response on this one.

Is there any update on Challenge 1?

As for #2 I have entered a UserVoice.
https://officespdev.uservoice.com/forums/224641-feature-requests-and-feedback/suggestions/34345651-add-in-developers-automated-testing-platform

@Reezaali Reezaali assigned davidchesnut and unassigned Reezaali Nov 20, 2018
@kbrandl
Copy link
Contributor

kbrandl commented Nov 28, 2018

Transferring this issue from OfficeDev/office-js-docs into OfficeDev/office-js-docs-pr.

@kbrandl kbrandl transferred this issue from OfficeDev/office-js-docs Nov 28, 2018
@sancarn
Copy link

sancarn commented Feb 18, 2019

Open office has javascript capability Might be possible to build a OfficeJS wrapper for open office. That'd provide a pretty good testing ground for developers at least. Would also allow OfficeJS addons to run on both platforms which could be nice... 😛

@r0bb077
Copy link

r0bb077 commented Feb 19, 2019

I see people have mentioned that they have been mocking the Office.js file, I don't suppose they'd care to share some examples on how they did this as I'm struggling currently?

@mattcopas
Copy link

+1 - some guidelines/examples on unit testing would be much appreciated :)

@shirbr510
Copy link

I tried mocking Office with some sort of an office-js wrapper I've written but so far have failed in doing so (had my whole mocks working but my app couldn't work with Officejs properly)

Would also love to see some sort of example or documentation

@shirbr510
Copy link

I have ended up doing the following @ setupTests.js (I'm using jest if it matters):

import OfficeMock from './__mocks__/Office'

const Office = global.Office;

beforeEach(()=>{
    global.Office = OfficeMock
})

afterEach(()=>{
    global.Office = Office
})

the OfficeMock is a simple object mapping that fits the properties I need from Office.js

I do believe that a proper solution including some guidelines and examples is still necessary though...

@oleksii-pi
Copy link

oleksii-pi commented Apr 16, 2019

Test:

/* eslint-disable @typescript-eslint/no-angle-bracket-type-assertion */
/* eslint-disable @typescript-eslint/no-object-literal-type-assertion */
...

test("getCustomRange", async() => {
  const expectedRange = <Word.Range>{text: "TT"};

  const paragraph: Word.Paragraph = <Word.Paragraph>{
    text: "textTTtextTtextTTtextTTtext",
    search: function(_text) {
      return <Word.RangeCollection>{
        load: function() {
          return this;
        },
        items: [null, expectedRange]
      };
    },
    context: {
      sync: function() {
        return new Promise(accept => accept());
      }
    }
  };

  expect(await ...).toBe(
    expectedRange
  );
});

@oleksii-pi
Copy link

oleksii-pi commented Apr 17, 2019

const context = {
      sync: () => {new Promise(accept => accept());},
      document: {
        body: {
          paragraphs: []
        }
      }
    };
    (global as any).Word = {run: async callBack => await callBack(context)}; // rollback if needed

@Alex-ray
Copy link

Hello @kbrandl , I also need support for this.

My team is building a word add-in that helps lawyers create templates for their documents. Naturally reliability is our number one priority here. End to end integration tests are critical in our ability to ensure our add-in's reliability.

The actual work this might take from your team feels low in relation to the potential increase in quality and reliability of your add-ins.

@kbrandl
Copy link
Contributor

kbrandl commented Apr 25, 2019

Thanks for reaching out @Alex-ray and I can certainly appreciate the perspective you've shared. This work item is in the documentation backlog; @davidchesnut is there any update you can provide regarding status of this work item?

@kbrandl kbrandl added the Needs: attention 👋 Waiting on Microsoft to provide feedback label Aug 26, 2019
@ElizabethSamuel-MSFT ElizabethSamuel-MSFT added Status: in backlog Issue is being tracked in the backlog but timeline for resolution is unknown and removed Needs: attention 👋 Waiting on Microsoft to provide feedback labels Oct 28, 2019
@seifeldeen92
Copy link

@kb339 Have you reached any progress in that? i am stuck in the same spot

@RohilaDasari
Copy link

Is there any update on this?

@JensMadsen
Copy link

Any updates? We just deployed our add-in with a huge bug in it ... We need automated tests

@celownick
Copy link

Hi @davidchesnut , @kbrandl and folks,
can I kindly ask for any updates on this ? We are preparing strategy for add-in testing and answering questions mentioned by @LanceEa wound have a crucial meaning for our team.
Thank you

@JensMadsen
Copy link

Any updates? We need to know the timeline for this "feature". If you know that this will happen in 2025 let us know. In that case, we will invest time in developing e.g. Selenium click e2e tests. If you, on the other hand, are actually working on this an expect a release in 2020 we will probably wait for it :-) But without an update, it is difficult to make a strategic decision on what to do (we must have test for CI/CD and QA). Best Regards

@ludmilanesvitiy
Copy link

Hey team! We also waiting for some updates with examples for e2e testing of word add-in. Please, post some updates - when do you plan to create this documentation? Is it possible at all to make this type of automated testing to work with word add-in. If I can help you with something relative to it to speed up the issue resolving - just let me know.

@ElizabethSamuel-MSFT
Copy link
Contributor

Related issue: #1611

@JensMadsen
Copy link

Any news? @ElizabethSamuel-MSFT

@JensMadsen
Copy link

We have started building ui tests using Selenium. We currently only test Word online. We plan to use Winium to test Word desktop Windows. The tests run on Github Actions and run headless in incognito mode. Currently, we only have a setup for Chrome (chromedriver). If you plan to create a similar setup please note that:

  1. In order for the addin to load and function properly you have to change the default Selenium cookie settings when running in incognito mode:
chromeOptions.setUserPreferences({ 'profile.cookie_controls_mode': 0 })

We had some problems before we realized this because Word Online loads fine and you will be able to login, but the addin does not show up anywhere (ribbon, appsource, edit->my addins, etc).

  1. For stable tests you have to fix the window sizes by running:
await driver
      .manage()
      .window()
      .setSize(3000, 2133)

(setRect for newer versions of Selenium).

  1. The addin runs in a designated iframe which is nested in the application iframe. In order to interact with the addin you have to switch to the parent iframe and then locate and switch to the addin iframe:
// switch to top iframe
await driver.wait(until.ableToSwitchToFrame(0))
// locate addin iframe
const addinIframes = await driver.findElements(
      By.xpath('//iframe[contains(@title,"<name of your addin>")]'),
    )
await driver.wait(until.ableToSwitchToFrame(addinIframes[0]))

Hope this can be of some help. If you need more details please ask :-)

@piemart
Copy link

piemart commented Jan 28, 2021

Hi @JensMadsen,
Thanks for the info.
I would like to setup UI testing with Selenium on an Outlook Desktop add-in with edge chromium, but I'm struggling.
The add-in starts the localhost:3000/taskpane.html and after some user inputs, the taskpane window opens an external appdomain (listed in the manifext.xml file). Here I would like Selenium to take over, so I can select the desired elements by their css_selector.
Do you think this is possible? Some more details would be appreciated :-)

@JensMadsen
Copy link

Hi @piemart

No quite sure what you wnat to achieve, but it sounds similar to what I do. In metasteps:

  1. start in incognito mode (I use Chrome). Incognito more stable
  2. login to office.com (make sure you have a test user). you have to pass username and password
  3. I open an existing test document (I test Word)-> office opens in a new tab
  4. go to new tab
  5. Do your tests (I test a taskpane addin; the one that load on the side )

@abdulkareemnalband
Copy link

We would like to use https://github.com/microsoft/playwright to automate the testing of desktop addins.
If word addin will expose WebView2 instance over Chrome DevTools Protocol , then we could use
https://playwright.dev/docs/api/class-browsertype#browser-type-connect-over-cdp to attach to that and use playwright to write tests

@mitoihs
Copy link

mitoihs commented Jun 29, 2021

My coworker has recently set up Selenium for desktop Excel add-in. That's what he said:

Since a lot of people are struggling, I guess I can give some information. Please note that this is a whacky way of doing it, but AFAIK only way for now.

Webview2 (chromium edge) has support for Selenium: https://docs.microsoft.com/en-us/microsoft-edge/webview2/how-to/webdriver
It is possible to instrumentate desktop Office (in my case Excel) to start Webview2 and connect with Selenium. To instrumentate Excel, I've used WinAppDriver. It is hard to set up (notable problems are WinAppDriver being slow and not suitable to drive Excel over simple clicks, WinAppDriver requiring Appium requiring Selenium v3, but Webview2 needs Selenium v4 - I've overcome this by using multiple projects that only interface externally).
If all goes correctly you will end up with WinAppDriver application that can start Office and connect Webview2, and Selenium that can drive add-in. It is hard to control the actual application, but for that purpose, I use OfficeJs via Selenium function ExecuteScript.

TL;DR: Possible to use Desktop Office with Selenium, very hard to set up.

@ferrykranenburgcw
Copy link

We would like to test our add-in in Outlook web using Playwright. But it looks very hard to just activate an e-mail and then activate our add-in inside Outlook. We also tried another way to test our code in Javascript only (using the Jest test framework) but we are missing bits an pieces from the Office.js framework to make it all work.

Any documentation about unit testing and integration testing add-in would be very welcome and helpful.

@AlexJerabek
Copy link
Collaborator

Hi everyone,

We appreciate all the discussion and feedback requesting testing solutions for Office Add-ins and Office JS. Unfortunately, there are no automated or unit testing solutions that we provide at this time. We're working on a solution, but have no concrete implementation or release details to share at this time. We realize this is an unsatisfying answer, but we feel it’s better to be clear about the support story, rather than to create false expectations.

We are moving the discussion to M365 Developer Platform Ideas. For the primary concern around standardized testing, please upvote and comment on this idea: Standard way to unit test and mock office-js API. For the other related requests that have accumulated on this issue, we also ask that you take your requests to the M365 Developer Platform Ideas and submit them as ideas (or upvote existing ones). This will help us prioritize things in an orderly fashion.

If there are other issues about the documentation that have been raised in this thread but were not addressed, please create a new issue to track each specific problem.

Thank you.

@lafe
Copy link

lafe commented Sep 6, 2021

@AlexJerabek I think, this is a very unsatisfying approach on how to deal with this issue. It has 42 comments and has been open for almost four years. By creating a new issue on a different platform, the complete history is gone (the new idea does not even reference this GitHub issue) and it appears as a new request. However, it is not a new customer request but an RFC from Microsoft to the community. Instead of posting it here and engaging directly with the affected developers, it is moved to a different platform and poses some questions. To me it appears, that Microsoft does not want to deal with this issue and moving it to a different platform buys time. Furthermore, if there is no feedback, you can internally point to the new idea and say "we tried but there seems to be no interest".

Furthermore, the new platform seems to be one where ideas go to die: currently, there are 217 ideas with the state "New" but only 4 ideas that have a different state (2 in "Working on it" and 2 in "Completed"). The ideas with different state where updated by their creators, so those are not community requests that were introduced through this platform. Even off-topic ideas like this one are not moderated. An actively used platform that is used for product planning and is actively monitored looks different...

I can somewhat understand, if you do not want to deal with this issue, as it is complex and might only be relevant for a subset of users. However, please be honest with it instead of pulling moves like this.

@giulianozor
Copy link

@lafe I agree with you. IMHO the project is (almost) dead and I see very little chances of it being maintained unless some major security issue is found. Unfortunately there are no alternatives AFAIK

@AlexJerabek
Copy link
Collaborator

Hi @lafe and @giulianozor,

I'm sorry that you felt unheard and disrespected by this action. We'll try to be better about how we handle these long-running issues in the future.

The goal of moving this issue and discussion was to be honest about the current state of testing Office-JS frameworks. The Developer Experience team (@igor-ribeiiro, @millerds, and @LouMM) is tracking this issue and trying to deliver a robust testing framework. However, there are no unit test frameworks or "mock" Office-JS frameworks that currently exist. There's nothing functional, but undocumented. Switching the issue from the documentation repo to Ideas is an effort to send the correct message about where the work is in the product lifecycle. We've added links to this thread (and the related unit testing thread) to the Idea, so everyone's contributions here are not lost or forgotten.

Thank for you also sharing the concerns about Tech Community. For the sake of transparency, I'll say the move to that platform was recent and sudden after needing to move from UserVoice. There are internal habits and processes that need to be strengthened. We need to do better as a team in monitoring Tech Community Ideas if we are going to recommend our developers go there to share their needs. Tagging @ElizabethSamuel-MSFT to help figure out how our team can increase their efficacy there.

Please know that your needs are heard and advocated for, beyond the vote count in Tech Community. I can say on behalf of myself and the other documentation writers that we want you to succeed in building on the Office Platform. The moment this testing work is in a usable state, we'll share it with everyone following this thread, demo it on the community call, and work with the community to ensure the documentation is helpful and useful.

@Rick-Kirkham
Copy link
Contributor

@lafe @giulianozor We now have documentation of unit testing and mocking the Office.js.

Unit testing in Office Add-ins

@rootlocus22
Copy link

Is there a Cypress like solution for Office Add In, we need to prepare an end to end integration testing framework to automate our flows?

@igor-ribeiiro
Copy link

Is there a Cypress like solution for Office Add In, we need to prepare an end to end integration testing framework to automate our flows?

Unfortunately, we currently don't have any end-to-end testing framework. So we don't have a Cypress-like solution at the current time.

@Rinzwind
Copy link

As a few others have commented here on using Selenium WebDriver for UI testing add-ins: one problem we’re bumping into is that Excel on office.com pops up a dialog saying “Looking to share a section of your workbook?” (see screenshot below) which covers the top part of our add-in’s task pane, so when a test tries to click an element there it gets an ‘element click intercepted’ error. As the dialog doesn’t seem to pop up consistently at the same point in the tests, I’m not quite sure yet how to deal with it. I’m wondering whether others have bumped into this and can offer advice? More generally: is there a more suitable place than this issue to ask questions about UI testing add-ins?

Dialog Share Section of Workbook

@AlexJerabek
Copy link
Collaborator

AlexJerabek commented Apr 3, 2023

is there a more suitable place than this issue to ask questions about UI testing add-ins?

@Rinzwind, Stack Overflow is the best way to get answers from others in the same situation. Stack's great for the intersection of technologies, as opposed to pure Office.js questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: add-ins Feedback on add-ins content that's relevant to multiple Office hosts. Status: in backlog Issue is being tracked in the backlog but timeline for resolution is unknown Type: doc request Request for new documentation or updates/enhancements to existing documentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.